Fix Java 11 Example PreCommit GItHub Action #28291

Abacn · 2023-09-02T23:12:47Z

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2023-09-02T23:38:30Z

Codecov Report

Merging #28291 (e28e958) into master (6d5048e) will decrease coverage by 0.04%.
Report is 7 commits behind head on master.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #28291      +/-   ##
==========================================
- Coverage   72.36%   72.32%   -0.04%     
==========================================
  Files         680      680              
  Lines      100263   100281      +18     
==========================================
- Hits        72557    72532      -25     
- Misses      26130    26178      +48     
+ Partials     1576     1571       -5

Flag	Coverage Δ
python	`82.89% <ø> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 12 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-09-03T00:19:08Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Abacn · 2023-09-03T01:57:39Z

Run Java PreCommit

damccorm · 2023-09-04T23:44:47Z

@celeste-zeng could you please take a look?

celeste-zeng · 2023-09-05T06:13:00Z

Run Java_Examples_Dataflow_ARM PostCommit 17

celeste-zeng · 2023-09-05T06:39:23Z

Run Java_Examples_Dataflow_ARM PostCommit 11

celeste-zeng

Actually, this will cause PostCommit Java11 Dataflow ARM to fail because error: option -source cannot be used together with --release and error: option -target cannot be used together with --release.

Abacn · 2023-09-05T16:17:05Z

I see, https://github.com/apache/beam/actions/runs/6081096373/job/16496105989 it appears at some place both --release and -source -target are set

celeste-zeng · 2023-09-05T16:57:49Z

Run Java_Examples_Dataflow_ARM PostCommit 11

celeste-zeng · 2023-09-05T17:53:44Z

Run Java_Examples_Dataflow_ARM PostCommit 11

celeste-zeng · 2023-09-05T18:27:01Z

Run Java_Examples_Dataflow_ARM PostCommit 17

Abacn · 2023-09-05T21:27:45Z

Run Java_Examples_Dataflow_ARM PostCommit

Abacn · 2023-09-05T21:27:52Z

Run Java_Examples_Dataflow_ARM PostCommit 11

Abacn · 2023-09-05T21:27:56Z

Run Java_Examples_Dataflow_ARM PostCommit 17

Abacn · 2023-09-05T21:42:04Z

Run SQL_Java17 PreCommit

Abacn · 2023-09-05T21:43:14Z

Run Java_Examples_Dataflow_ARM PostCommit 17

Abacn · 2023-09-05T21:43:28Z

Run Java_Examples_Dataflow_ARM PostCommit 11

Abacn · 2023-09-05T21:43:33Z

Run Java_Examples_Dataflow_ARM PostCommit 8

Abacn · 2023-09-05T21:53:42Z

Run Java_Examples_Dataflow_Java11 PreCommit

https://github.com/apache/beam/actions/runs/6090134291/job/16524397838

Abacn · 2023-09-05T21:56:28Z

looks like github comment event will kill ongoing phrase jobs. In addition, if the job hasn't been triggered by commit, the job status is not updated to the PR page.

Abacn · 2023-09-05T21:57:03Z

Run Java_Examples_Dataflow_ARM PostCommit 11

Update: it even gets cancelled without another comment (likely issue comment in another PR?)

Abacn · 2023-09-05T23:03:22Z

Run Java_Examples_Dataflow_ARM PostCommit 11

passed on java11: https://github.com/apache/beam/actions/runs/6090627647/job/16525833339

Abacn · 2023-09-05T23:54:42Z

Run Java_Examples_Dataflow_ARM PostCommit 17

passed on Java17: https://github.com/apache/beam/actions/runs/6090943543/job/16526707579

Abacn · 2023-09-06T00:32:58Z

Run Java_Examples_Dataflow_Java11 PreCommit

Well, it returns to the behavior at master: https://github.com/apache/beam/actions/runs/6091222107/job/16527475658

Error happens on Dataflow side:

"java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
	at org.apache.beam.sdk.io.TextSource$TextBasedReader.readDefaultLine(TextSource.java:263)
	at org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:222)
	at org.apache.beam.sdk.io.TextSource$TextBasedReader.startReading(TextSource.java:181)
	at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:491)
	at org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:252)
	at org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.advance(UnboundedReadFromBoundedSource.java:474)
	at org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.access$300(UnboundedReadFromBoundedSource.java:452)
	at org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$Reader.advance(UnboundedReadFromBoundedSource.java:304)
	at org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$Reader.start(UnboundedReadFromBoundedSource.java:297)
	at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:816)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:381)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:211)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:169)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
	at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1433)
	at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:155)
	at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$4.run(StreamingDataflowWorker.java:1056)
	at org.apache.beam.runners.dataflow.worker.util.BoundedQueueExecutor.lambda$executeLockHeld$0(BoundedQueueExecutor.java:181)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

it appears Dataflow still uses Java8 to run the pipeline? Even though "userAgent
Apache_Beam_SDK_for_Java/2.51.0-SNAPSHOT(JRE_11_environment)"

Abacn · 2023-09-06T04:29:04Z

Run Java_Examples_Dataflow_Java11 PreCommit

Fixed: https://github.com/apache/beam/actions/runs/6092731200

Abacn · 2023-09-06T14:00:19Z

PTAL @celeste-zeng

Abacn · 2023-09-06T14:08:00Z

Issues revealed in detail

duplicate compiler flags have always been added to TestCompileJava under Java11/17, however, because the latter option will overwrite the previous one, it did not show error.
Fix Java 11 Example PreCommit GItHub Action #28291 changed all occurrance of --release to -source X -target X. They are not equivalent, actually: ((ref)[https://stackoverflow.com/questions/43102787/what-is-the-release-flag-in-the-java-9-compiler]). In particular, here it has to be --release:

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Line 1095 in e28e958

options.compilerArgs += ['--release', '8']

Otherwise, using Java9+ compile beam main code then run on Dataflow runner will see the error in #28291 (comment) (current master java11 example precommit error)

celeste-zeng

LGTM, thanks for digging into this!

Partially revert apache#28234 to fix Java 11 Example PreCommit

2645e3a

github-actions bot added the build label Sep 2, 2023

github-actions bot added the Next Action: Reviewers label Sep 3, 2023

celeste-zeng approved these changes Sep 5, 2023

View reviewed changes

celeste-zeng reviewed Sep 5, 2023

View reviewed changes

Dedup compile args

ca12436

Abacn force-pushed the fixjava11precommit branch from 46045ff to ca12436 Compare September 5, 2023 16:53

fix typo

2aae0bc

Abacn added 2 commits September 5, 2023 15:12

Fix compilerArgs addAll

1ca7a88

Move compileAndRunTestsWithJavaX after errorprone has set up

de8e680

Abacn changed the title ~~Partially revert #28234 to fix Java 11 Example PreCommit~~ Fix Java 11 Example PreCommit GItHub Action Sep 5, 2023

Still use --release for java8 option

e28e958

celeste-zeng approved these changes Sep 6, 2023

View reviewed changes

Abacn merged commit 73a9c5f into apache:master Sep 6, 2023

Abacn deleted the fixjava11precommit branch September 6, 2023 17:01

This was referenced Sep 28, 2023

[Failing Test]: beam_PostCommit_Java_Jpms on Java17 permared #28724

Closed

Fix Jpms test #28726

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Java 11 Example PreCommit GItHub Action #28291

Fix Java 11 Example PreCommit GItHub Action #28291

Abacn commented Sep 2, 2023

codecov bot commented Sep 2, 2023 •

edited

Loading

github-actions bot commented Sep 3, 2023

Abacn commented Sep 3, 2023

damccorm commented Sep 4, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng left a comment •

edited

Loading

Abacn commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 6, 2023 •

edited

Loading

Abacn commented Sep 6, 2023 •

edited

Loading

Abacn commented Sep 6, 2023

Abacn commented Sep 6, 2023

celeste-zeng left a comment

Fix Java 11 Example PreCommit GItHub Action #28291

Fix Java 11 Example PreCommit GItHub Action #28291

Conversation

Abacn commented Sep 2, 2023

GitHub Actions Tests Status (on master branch)

codecov bot commented Sep 2, 2023 • edited Loading

Codecov Report

github-actions bot commented Sep 3, 2023

Abacn commented Sep 3, 2023

damccorm commented Sep 4, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng left a comment • edited Loading

Choose a reason for hiding this comment

Abacn commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

celeste-zeng commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023 • edited Loading

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023 • edited Loading

Abacn commented Sep 5, 2023 • edited Loading

Abacn commented Sep 5, 2023 • edited Loading

Abacn commented Sep 6, 2023 • edited Loading

Abacn commented Sep 6, 2023 • edited Loading

Abacn commented Sep 6, 2023

Abacn commented Sep 6, 2023

celeste-zeng left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 2, 2023 •

edited

Loading

celeste-zeng left a comment •

edited

Loading

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 5, 2023 •

edited

Loading

Abacn commented Sep 6, 2023 •

edited

Loading

Abacn commented Sep 6, 2023 •

edited

Loading