[#29772][prism] Handle EventTime Timers.

[lostluck]

Updates Prism to support EventTime Timers.

Fixes a bug in Go SDK timer handling unset watermark holds would have the wrong default hold. They should default to the firing time.

Timer implementation tracked in #29772.

Largely handles event time timers by treating them as data elements in the processing, with the goal of minimizing unique handling of timers vs data.
Timers with the same event time must always sort after data elements with the same event time, so they can be reasonably be consolidated when firing.

Since only a single timer can be set for a [window, user key, timerfamily, tag] tuple we track the last set EventTime + Hold. This avoids scanning the element queue for a key every time a timer is written at the expense that more timers are retained in memory (a "spend RAM to save CPU" trade off). This requires managing the remaining "pending elements" properly as well, as a timer may be written, but it may not affect size of the pending queue.

WatermarkHolds are tracked on a per Hold basis. Each stage maintains a heap of their holds, and a map of how many timers are using that hold. When the count goes to 0, we remove the hold from the map, and the heap. This approach avoids tracking holds on a per timer basis, which would mean a much larger hold heap.

Aside: A general improvement that could be made to the Element Manager would be to "intern" the user key bytes, tags, and various other strings when persisting the data. This would avoid retaining several copies of each set of bytes in memory between stages (this is a "spend CPU to save RAM" trade off). There's also likely opportunity to clean up "leaked" tails of large byte buffers when a key prefix is being held (a similar CPU for RAM tradeoff).

---

Deferred to a later PR, related to EventTime timers:

• Handling Cleared timers correctly.
  • Fired timers are cleared, but a user clearing a timer (eg. because a batch is determined complete, so a GC timer isn't needed) isn't tested yet. It's expected that such a timer will simply fire.
• Handling OnWindowExpiration, because it's not implemented in the Go SDK yet, so it's hard to test.

Both are likely to be handled in the process of cleaning up prism for the Java and Python Validates Runner tests.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[codecov]

Codecov Report

Attention: 123 lines in your changes are missing coverage. Please review.

Comparison is base (6751a17) 38.24% compared to head (3af2120) 38.39%.
Report is 53 commits behind head on master.

Files	Patch %	Lines
...o/pkg/beam/runners/prism/internal/engine/timers.go	66.03%	33 Missing and 3 partials ⚠️
...am/runners/prism/internal/engine/elementmanager.go	88.16%	25 Missing and 4 partials ⚠️
...o/pkg/beam/runners/prism/internal/worker/bundle.go	46.51%	22 Missing and 1 partial ⚠️
sdks/go/pkg/beam/runners/prism/internal/stage.go	70.42%	16 Missing and 5 partials ⚠️
...o/pkg/beam/runners/prism/internal/worker/worker.go	14.28%	11 Missing and 1 partial ⚠️
...m/runners/prism/internal/jobservices/management.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #29900      +/-   ##
==========================================
+ Coverage   38.24%   38.39%   +0.14%     
==========================================
  Files         696      697       +1     
  Lines      101881   102293     +412     
==========================================
+ Hits        38963    39274     +311     
- Misses      61306    61391      +85     
- Partials     1612     1628      +16     

Flag	Coverage Δ
go	54.19% <74.79%> (+0.23%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label go.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[lostluck]

R: @damondouglas

And in the meantime, I'll validate the RC.

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[lostluck]

Friendly Ping.

[damondouglas]

Overall I had trouble understanding the code changes of this PR.

[christ]

Did you mean to assign this to me?

[lostluck]

@christ Sorry about that. Something went haywire with keyboard shortcuts while I was replying to a comment. It must have done the assignement then. TIL there are GH keyboard shortcuts for assignee-asigning....

[christ]

No worries, just didn't want it lost in the ether.

[damondouglas]

LGTM

[lostluck]

Thanks for the review!