Create Avro extension for Java SDK #24294

aromanenko-dev · 2022-11-21T17:50:22Z

Create a dedicated extension for Java SDK to support Avro format.

This PR mostly includes copy-pasted Avro-related code from sdks/java/core with the same package structure and it introduces almost no changes to "core" - only several access modifiers are relaxed to be able to use from other packages. It should not break any other user code.

This is the first step to use "Avro from extension" instead of "Avro from core".

Closes #24293

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2022-11-21T18:32:31Z

Codecov Report

Merging #24294 (150877e) into master (820839f) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #24294      +/-   ##
==========================================
- Coverage   73.27%   73.25%   -0.02%     
==========================================
  Files         725      725              
  Lines       97444    97444              
==========================================
- Hits        71403    71385      -18     
- Misses      24691    24709      +18     
  Partials     1350     1350

Flag	Coverage Δ
python	`82.83% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../python/apache_beam/testing/test_stream_service.py	`88.09% <0.00%> (-4.77%)`	⬇️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (-2.44%)`	⬇️
...ks/python/apache_beam/runners/worker/data_plane.py	`87.57% <0.00%> (-1.70%)`	⬇️
.../python/apache_beam/transforms/periodicsequence.py	`97.01% <0.00%> (-1.50%)`	⬇️
...che_beam/runners/interactive/interactive_runner.py	`90.50% <0.00%> (-1.27%)`	⬇️
sdks/python/apache_beam/io/localfilesystem.py	`90.97% <0.00%> (-0.76%)`	⬇️
...eam/runners/portability/fn_api_runner/execution.py	`92.49% <0.00%> (-0.64%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.54% <0.00%> (-0.13%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

aromanenko-dev · 2022-12-20T16:37:09Z

R: @mosche
FYI: CC: @kennknowles @lukecwik @reuvenlax @robertwb
ptal

github-actions · 2022-12-20T17:05:08Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @Abacn for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

mosche · 2022-12-21T13:19:08Z

👀

mosche · 2022-12-21T13:26:42Z

Run Spotless PreCommit

mosche · 2022-12-21T13:29:23Z

Closes #24293

This PR doesn't close the ticket yet, right?

sdks/java/extensions/avro/build.gradle

aromanenko-dev · 2022-12-21T15:08:36Z

@mosche

This PR doesn't close the ticket yet, right?

It does since it's just a sub-task of #24292

mosche

Some questions, pls have a look @aromanenko-dev
I also found a couple of copy-paste issues. I've checked the diff for all newly committed files. Looks fine except for the cases mentioned below.

find sdks/java -name AvroCoder.java | xargs diff --color

$ grep -r -E 'org\.apache\.beam\.sdk\.(schemas|io)(\.\w+)*\.\w*Avro' sdks/java/extensions/avro

sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java:        new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java:        new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java:        new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java:        new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java:        new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java:        new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java:        new org.apache.beam.sdk.schemas.AvroRecordSchema()
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:    private static <T> org.apache.beam.sdk.io.AvroSource<T> createSource(
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:        org.apache.beam.sdk.io.AvroSource.@Nullable DatumReaderFactory<T> readerFactory) {
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:      org.apache.beam.sdk.io.AvroSource<?> source =
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:          org.apache.beam.sdk.io.AvroSource.from(filepattern)
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:          ? (org.apache.beam.sdk.io.AvroSource<T>) source.withSchema(schema)
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:    abstract org.apache.beam.sdk.io.AvroSource.@Nullable DatumReaderFactory<T>
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:          org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> factory);
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:        org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> factory) {
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:    private final org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> readerFactory;
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:        org.apache.beam.sdk.io.AvroSource.DatumReaderFactory<T> readerFactory) {
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java:                org.apache.beam.sdk.io.AvroSource.from(getFilepattern())
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java: * A specialization of {@link DynamicDestinations} for {@link org.apache.beam.sdk.io.AvroIO}. In
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java:   * Return a {@link org.apache.beam.sdk.io.AvroSink.DatumWriterFactory} for a given destination. If

sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSink.java

sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java

...sions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java

...tensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java

...xtensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/AvroSchemaTest.java

...sions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java

aromanenko-dev · 2022-12-26T12:45:55Z

Run Java_GCP_IO_Direct PreCommit

aromanenko-dev · 2022-12-26T14:04:58Z

Run Java_Pulsar_IO_Direct PreCommit

aromanenko-dev · 2022-12-26T15:30:02Z

@mosche Thanks for review, I believe I addressed all your comments, PTAL

mosche · 2022-12-28T09:05:03Z

👀

mosche

Thanks @aromanenko-dev, looking good now 🎄 ✨
Feel free to merge!

github-actions bot added build extensions java labels Nov 21, 2022

aromanenko-dev changed the title ~~Create an Avro extension for Java SDK~~ [WIP] Create an Avro extension for Java SDK Nov 21, 2022

aromanenko-dev marked this pull request as draft November 21, 2022 17:51

aromanenko-dev force-pushed the extensions_avro branch 2 times, most recently from 6fc5e88 to 67240a7 Compare December 20, 2022 16:26

aromanenko-dev marked this pull request as ready for review December 20, 2022 16:28

aromanenko-dev changed the title ~~[WIP] Create an Avro extension for Java SDK~~ Create Avro extension for Java SDK Dec 20, 2022

github-actions bot added the Next Action: Reviewers label Dec 20, 2022

mosche reviewed Dec 21, 2022

View reviewed changes

sdks/java/extensions/avro/build.gradle Show resolved Hide resolved

mosche requested changes Dec 21, 2022

View reviewed changes

aromanenko-dev force-pushed the extensions_avro branch from 67240a7 to 06dd451 Compare December 21, 2022 15:33

mosche reviewed Dec 22, 2022

View reviewed changes

...sions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java Outdated Show resolved Hide resolved

mosche reviewed Dec 22, 2022

View reviewed changes

...sions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java Outdated Show resolved Hide resolved

aromanenko-dev force-pushed the extensions_avro branch from 06dd451 to 8ec7419 Compare December 22, 2022 17:05

aromanenko-dev added 3 commits December 26, 2022 13:47

[apache#24292] Create Avro extension for Java SDK

873e0d0

Address the review comments

8b532db

Apply apache#24454 changes

150877e

aromanenko-dev force-pushed the extensions_avro branch from 8ec7419 to 150877e Compare December 26, 2022 13:44

mosche self-requested a review December 28, 2022 09:48

mosche approved these changes Dec 28, 2022

View reviewed changes

aromanenko-dev merged commit bab773f into apache:master Dec 28, 2022

aromanenko-dev deleted the extensions_avro branch December 28, 2022 14:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Avro extension for Java SDK #24294

Create Avro extension for Java SDK #24294

aromanenko-dev commented Nov 21, 2022 •

edited

Loading

codecov bot commented Nov 21, 2022 •

edited

Loading

aromanenko-dev commented Dec 20, 2022 •

edited

Loading

github-actions bot commented Dec 20, 2022

mosche commented Dec 21, 2022

mosche commented Dec 21, 2022

mosche commented Dec 21, 2022

aromanenko-dev commented Dec 21, 2022

mosche left a comment •

edited

Loading

aromanenko-dev commented Dec 26, 2022

aromanenko-dev commented Dec 26, 2022

aromanenko-dev commented Dec 26, 2022

mosche commented Dec 28, 2022

mosche left a comment

Create Avro extension for Java SDK #24294

Create Avro extension for Java SDK #24294

Conversation

aromanenko-dev commented Nov 21, 2022 • edited Loading

GitHub Actions Tests Status (on master branch)

codecov bot commented Nov 21, 2022 • edited Loading

Codecov Report

aromanenko-dev commented Dec 20, 2022 • edited Loading

github-actions bot commented Dec 20, 2022

mosche commented Dec 21, 2022

mosche commented Dec 21, 2022

mosche commented Dec 21, 2022

aromanenko-dev commented Dec 21, 2022

mosche left a comment • edited Loading

Choose a reason for hiding this comment

aromanenko-dev commented Dec 26, 2022

aromanenko-dev commented Dec 26, 2022

aromanenko-dev commented Dec 26, 2022

mosche commented Dec 28, 2022

mosche left a comment

Choose a reason for hiding this comment

aromanenko-dev commented Nov 21, 2022 •

edited

Loading

codecov bot commented Nov 21, 2022 •

edited

Loading

aromanenko-dev commented Dec 20, 2022 •

edited

Loading

mosche left a comment •

edited

Loading