Stop emitting upon truncate in Java PeriodicSequence #25716
Conversation
| @@ -207,7 +213,7 @@ public ProcessContinuation processElement( | |||
|
|
|||
| boolean claimSuccess = true; | |||
|
|
|||
| estimator.setWatermark(Instant.ofEpochMilli(restriction.getFrom())); | |||
| estimator.setWatermark(Instant.ofEpochMilli(nextOutput)); | |||
There was a problem hiding this comment.
This change is made in accordance with Python implementation. Currently the watermark will be set back to the initial timestamp (which violates the monotonically increasing) though it will be set back in the while clause.
|
During testing it is also found that runner v2 calls newTracker as much as 5x than runner v1, even though the runner v2 job has 4 CPU core and runner v1 job only has two. This may have performance impacts since newTracer is used in every splittable DoFn. runner v1 log:
|
|
R: @damccorm |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
|
Run Java_IOs_Direct PreCommit |
|
Java PreCommit known flake KafkaIO precommit known flake: testWatermarkUpdateWithSparseMessages #25261 |
|
Run Java PreCommit |
|
Run Java_Kafka_IO_Direct PreCommit |
Fixes #22776. Java counterpart of the python fix: #23765
Tested draining successful on Dataflow runner v2. JobId:
2023-03-03_10_53_37-6111128436657959155However Dataflow runner v1 still not working: JobId:
2023-03-03_10_54_38-5379949162309256925checked that on runner v1 the truncate function never gets called after hitting the drain button. This may be a feature gap that Dataflow runner v1 does not support drain splittable DoFn souce.
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.