Initial start of a yaml-based declarative way of building pipelines. #24667

robertwb · 2022-12-14T21:05:21Z

E.g.

pipeline:
  - type: chain
    transforms:
      - type: ReadFromText
        args:
         file_pattern: "wordcount.yaml"
      - type: PyFlatMap
        fn: "import re\nlambda line: re.findall('\\w+', line)"
      - type: PyMap
        fn: "str.lower"
      - type: PyTransform
        name: Count
        constructor: "apache_beam.transforms.combiners.Count.PerElement"
      - type: PyMap
        fn: str
      - type: WriteToText
        file_path_prefix: "counts.txt"

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2022-12-14T21:31:29Z

Codecov Report

Merging #24667 (b78009c) into master (6bbc396) will decrease coverage by 0.21%.
The diff coverage is 69.21%.

@@            Coverage Diff             @@
##           master   #24667      +/-   ##
==========================================
- Coverage   73.35%   73.15%   -0.21%     
==========================================
  Files         719      732      +13     
  Lines       97137    98093     +956     
==========================================
+ Hits        71257    71762     +505     
- Misses      24532    24983     +451     
  Partials     1348     1348

Flag	Coverage Δ
python	`82.56% <69.21%> (-0.43%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/yaml/main.py	`0.00% <0.00%> (ø)`
sdks/python/apache_beam/transforms/external.py	`78.74% <50.00%> (-0.14%)`	⬇️
sdks/python/apache_beam/yaml/yaml_provider.py	`63.47% <63.47%> (ø)`
sdks/python/apache_beam/yaml/yaml_transform.py	`80.60% <80.60%> (ø)`
sdks/python/apache_beam/yaml/__init__.py	`100.00% <100.00%> (ø)`
sdks/python/apache_beam/utils/python_callable.py	`86.88% <0.00%> (-11.16%)`	⬇️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (-2.44%)`	⬇️
...apache_beam/typehints/native_type_compatibility.py	`82.82% <0.00%> (-2.34%)`	⬇️
...python/apache_beam/runners/worker/worker_status.py	`75.33% <0.00%> (-1.34%)`	⬇️
sdks/python/apache_beam/internal/metrics/metric.py	`93.00% <0.00%> (-1.00%)`	⬇️
... and 31 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

chamikaramj

Thanks. Just adding some initial fly by comments.

sdks/python/apache_beam/yaml/standard_providers.yaml

chamikaramj · 2022-12-14T23:32:45Z

sdks/python/apache_beam/yaml/yaml_provider.py

+    self._service = service
+    self._schema_transforms = None
+
+  def provided_transforms(self):


Should be "provided_transform_urns" ?

No, these aren't the URNs but the "friendly" names.

sdks/python/apache_beam/yaml/yaml_provider.py

robertwb

Updated per some of the feedback.

sdks/python/apache_beam/yaml/yaml_provider.py

sdks/python/apache_beam/yaml/standard_providers.yaml

github-actions · 2022-12-20T02:22:56Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @AnandInguva for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

github-actions · 2023-01-04T12:14:05Z

Reminder, please take a look at this pr: @AnandInguva

github-actions · 2023-01-09T12:13:45Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

damccorm · 2023-01-09T15:11:39Z

stop reviewer notifications

damccorm · 2023-01-09T15:12:00Z

(looks like Cham has been reviewing, I'll let him take the review side to completion)

github-actions · 2023-01-09T15:13:12Z

Stopping reviewer notifications for this pull request: requested by reviewer

chamikaramj

Thanks!

sdks/python/apache_beam/yaml/main.py

chamikaramj · 2023-01-18T23:04:52Z

sdks/python/apache_beam/yaml/main.py

+    pipeline_spec = yaml.load(
+        known_args.pipeline_spec, Loader=yaml_transform.SafeLineLoader)
+  else:
+    raise ValueError(


Seems like this check is not done at the correct location (it will be triggered if just known_args.pipeline_spec_file is set).

known_args.pipeline_spec is was populated from the file above in that case. But this didn't catch the case where both were set. Spelling it out more explicitly now.

sdks/python/apache_beam/yaml/yaml_provider.py

sdks/python/apache_beam/yaml/yaml_transform.py

chamikaramj · 2023-01-19T23:44:31Z

sdks/python/apache_beam/yaml/yaml_transform.py

+  def __init__(self, spec, providers={}):  # pylint: disable=dangerous-default-value
+    if isinstance(spec, str):
+      spec = yaml.load(spec, Loader=SafeLineLoader)
+    self._spec = spec


Can/should we validate the spec somehow before accepting it ?

I added a schema that we can validate against.

chamikaramj · 2023-01-20T00:07:03Z

sdks/python/apache_beam/yaml/yaml_provider.py

+        self._schema_transforms = []
+    urn = self._urns[type]
+    if urn in self._schema_transforms:
+      return external.SchemaAwareExternalTransform(urn, self._service, **args)


Noting that this will limit Yaml-based pipelines to portable runners (which is fine but should be documented).

True, but all available runners in Python are portable runners (and I'd be tempted to say that the other runners aren't "real" beam Runners :-). Is there somewhere specific you'd want this mentioned?

I don't have a good place here but this should go to Yaml runner documentation when we have it.

chamikaramj · 2023-01-20T00:12:06Z

sdks/python/apache_beam/yaml/yaml_transform_test.py

+          input:
+              elements: input
+          transforms:
+            - type: PyMap


Is there any meaning to deeper levels of nesting or should all transforms listed at the top level ?

This creates a composite transform. One could also do

YamlTransform( ''' type: PyMap name: Cube ''' )

to get a single transform. I agree the input here is a bit hacky, dropped a TODO to think about this.

chamikaramj · 2023-01-20T00:13:00Z

sdks/python/apache_beam/yaml/yaml_transform_test.py

+      assert_that(result, equal_to([1, 4, 9, 1, 8, 27]))
+
+  def test_chain_with_input(self):
+    with beam.Pipeline(options=beam.options.pipeline_options.PipelineOptions(


We should also add tests for other types of providers.

True, but that requires invoking the whole cross-langauge testing infrastructure framework, and I was hoping to run these as unit tests (mostly validating the parsing and construction).

This reverts commit 2362cda.

chamikaramj

Thanks. LGTM.

Seems like we might need to add more tests and documentation but those can be added in follow up PRs.

chamikaramj · 2023-01-23T20:05:16Z

sdks/python/apache_beam/yaml/yaml_provider.py

+        self._schema_transforms = []
+    urn = self._urns[type]
+    if urn in self._schema_transforms:
+      return external.SchemaAwareExternalTransform(urn, self._service, **args)


I don't have a good place here but this should go to Yaml runner documentation when we have it.

chamikaramj · 2023-01-23T20:13:17Z

sdks/python/apache_beam/yaml/yaml_provider.py

+      return constructor >> FullyQualifiedNamedTransform(
+          constructor, args, kwargs)
+
+  class Flatten(beam.PTransform):


I see. Let's add a comment here regarding needing the extra Flatten to support one or zero PCollections.

robertwb · 2023-01-24T17:47:48Z

Thanks!

Initial start of a yaml-based declarative way of building pipelines.

e2b6083

github-actions bot added the python label Dec 14, 2022

Dataframes fix.

c551801

chamikaramj reviewed Dec 14, 2022

View reviewed changes

robertwb added 2 commits December 19, 2022 16:32

Address some reviewer comments.

37e9ca0

Add some tests.

585989e

robertwb commented Dec 20, 2022

View reviewed changes

sdks/python/apache_beam/yaml/yaml_provider.py Show resolved Hide resolved

sdks/python/apache_beam/yaml/standard_providers.yaml Outdated Show resolved Hide resolved

robertwb marked this pull request as ready for review December 20, 2022 00:41

robertwb added 2 commits December 19, 2022 16:45

Attach licence to one-line functional import.

68e7df1

lint

b227761

github-actions bot added the Next Action: Reviewers label Dec 20, 2022

robertwb added 4 commits December 21, 2022 11:34

Make jar fetching lazy.

f8d4207

lint

865b6a7

lint

e92bd2a

isort

768a960

github-actions bot added the slow-review label Jan 4, 2023

github-actions bot removed the slow-review label Jan 9, 2023

chamikaramj requested changes Jan 20, 2023

View reviewed changes

robertwb added 5 commits January 20, 2023 10:18

address comments

0c31b89

add schema

3c1c1df

use json rather than yaml for schema

2362cda

Revert "use json rather than yaml for schema"

7ef6b9b

This reverts commit 2362cda.

A note about providers.

fe75890

robertwb added 2 commits January 20, 2023 12:26

lint

dda4b2c

lint

1d57bc1

chamikaramj reviewed Jan 23, 2023

View reviewed changes

robertwb added 2 commits January 23, 2023 16:11

lint, comments, annotations

86aef7a

isort

b78009c

robertwb merged commit 6b8f8c0 into apache:master Jan 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial start of a yaml-based declarative way of building pipelines. #24667

Initial start of a yaml-based declarative way of building pipelines. #24667

robertwb commented Dec 14, 2022

codecov bot commented Dec 14, 2022 •

edited

Loading

chamikaramj left a comment

chamikaramj Dec 14, 2022

robertwb Dec 15, 2022

robertwb left a comment

github-actions bot commented Dec 20, 2022

github-actions bot commented Jan 4, 2023

github-actions bot commented Jan 9, 2023

damccorm commented Jan 9, 2023

damccorm commented Jan 9, 2023

github-actions bot commented Jan 9, 2023

chamikaramj left a comment

chamikaramj Jan 18, 2023

robertwb Jan 20, 2023

chamikaramj Jan 19, 2023

robertwb Jan 20, 2023

chamikaramj Jan 20, 2023

robertwb Jan 20, 2023

chamikaramj Jan 23, 2023

chamikaramj Jan 20, 2023

robertwb Jan 20, 2023

chamikaramj Jan 20, 2023

robertwb Jan 20, 2023

chamikaramj left a comment

chamikaramj Jan 23, 2023

chamikaramj Jan 23, 2023

robertwb commented Jan 24, 2023

Initial start of a yaml-based declarative way of building pipelines. #24667

Initial start of a yaml-based declarative way of building pipelines. #24667

Conversation

robertwb commented Dec 14, 2022

GitHub Actions Tests Status (on master branch)

codecov bot commented Dec 14, 2022 • edited Loading

Codecov Report

chamikaramj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertwb left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 20, 2022

github-actions bot commented Jan 4, 2023

github-actions bot commented Jan 9, 2023

damccorm commented Jan 9, 2023

damccorm commented Jan 9, 2023

github-actions bot commented Jan 9, 2023

chamikaramj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chamikaramj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertwb commented Jan 24, 2023

codecov bot commented Dec 14, 2022 •

edited

Loading