Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle #19957

Closed
damccorm opened this issue Jun 4, 2022 · 0 comments · Fixed by #24321
Closed

Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle #19957

damccorm opened this issue Jun 4, 2022 · 0 comments · Fixed by #24321
Labels
bug dataflow P3 runners
Milestone
2.44.0 Release

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 4, 2022

Let's suppose we have the following situation:

  • statful ParDo with two timers - timerA and timerB
  • timerA is set for window.maxTimestamp() **** 1
  • timerB is set anywhere between <windowStart, windowEnd), let's denote that timerB.timestamp
  • input watermark moves to BoundedWindow.TIMESTAMP_MAX_VALUE

Then the order of timers is as follows (correct):

  • timerB
  • timerA

But, if timerB sets another timer (say for timerB.timestamp **** 1), then the order of timers will be:

  • timerB (timerB.timestamp)
  • timerA (BoundedWindow.TIMESTAMP_MAX_VALUE)
  • timerB (timerB.timestamp **** 1)

Which is not ordered by timestamp. The reason for this is that when the input watermark update is evaluated, the WatermarkManager,extractFiredTimers() will produce both timerA and timerB. That would be correct, but when timerB sets another timer, that breaks this.

Imported from Jira BEAM-8543. Original Jira may contain additional context.
Reported by: janl.
@lukecwik lukecwik mentioned this issue Nov 22, 2022
Merged
4 tasks
lukecwik added a commit to lukecwik/incubator-beam that referenced this issue Nov 22, 2022
@lukecwik
Enable streaming runner v2 tests that were forgotten to be enabled.
89401b2 
These tests are run internally on Google and have been passing for quite some time.

Fixes apache#19957
Fixes apache#20726
Fixes apache#20601
Fixes apache#20734
lukecwik added a commit that referenced this issue Nov 24, 2022
@lukecwik
Enable streaming runner v2 tests that were forgotten to be enabled. (#…
3b7e181 
…24321)

* Enable streaming runner v2 tests that were forgotten to be enabled.

These tests are run internally on Google and have been passing for quite some time.

Fixes #19957
Fixes #20726
Fixes #20601
Fixes #20734

* Explicitly enable streaming engine for runner based autosharding test
@github-actions github-actions bot added this to the 2.44.0 Release milestone Nov 24, 2022
ruslan-ikhsan pushed a commit to ruslan-ikhsan/beam that referenced this issue Nov 30, 2022
@lukecwik @ruslan-ikhsan
Enable streaming runner v2 tests that were forgotten to be enabled. (a…
445ec8d 
…pache#24321)

* Enable streaming runner v2 tests that were forgotten to be enabled.

These tests are run internally on Google and have been passing for quite some time.

Fixes apache#19957
Fixes apache#20726
Fixes apache#20601
Fixes apache#20734

* Explicitly enable streaming engine for runner based autosharding test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug dataflow P3 runners
Projects
None yet
Milestone
2.44.0 Release
Development

Successfully merging a pull request may close this issue.

Enable streaming runner v2 tests that were forgotten to be enabled. lukecwik/incubator-beam
1 participant
@damccorm