#20812 handle @RequiresStableInput in portable flink #22889

je-ik · 2022-08-25T12:08:45Z

Fixes #20812
Fixes #21333

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

je-ik · 2022-08-25T13:01:42Z

R: @ibzib @dmvk

github-actions · 2022-08-25T15:08:20Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

github-actions · 2022-08-25T15:08:22Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

je-ik · 2022-08-29T08:19:47Z

assign set of reviewers

je-ik · 2022-08-30T15:14:28Z

Run Java Flink PortableValidatesRunner Streaming

je-ik · 2022-09-05T09:36:24Z

Run Flink ValidatesRunner

je-ik · 2022-09-05T09:38:30Z

Run Java Flink PortableValidatesRunner Streaming

je-ik · 2022-09-05T11:34:23Z

Run Flink ValidatesRunner

je-ik · 2022-09-05T11:34:32Z

Run Java Flink PortableValidatesRunner Streaming

je-ik · 2022-09-05T12:36:50Z

Run Flink ValidatesRunner

je-ik · 2022-09-05T12:36:57Z

Run Java Flink PortableValidatesRunner Streaming

je-ik · 2022-09-05T14:19:26Z

R: @lukecwik

je-ik · 2022-09-27T08:09:49Z

@aaltay @robertwb @kennknowles Looks I'm unable to find a reviewer for this one. It fixes correctness bug in Flink, cau you please help find one? Thanks. 👍

aaltay · 2022-09-27T16:03:28Z

@kileys - Would you be able to review this? It is a medium sized change and does not require too much flink runner context.

codecov · 2022-12-14T16:17:28Z

Codecov Report

Merging #22889 (0694831) into master (0f423dd) will increase coverage by 7.98%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #22889      +/-   ##
==========================================
+ Coverage   65.20%   73.19%   +7.98%     
==========================================
  Files         731      734       +3     
  Lines       97960    98897     +937     
==========================================
+ Hits        63878    72390    +8512     
+ Misses      32727    25152    -7575     
  Partials     1355     1355

Flag	Coverage Δ
python	`82.81% <ø> (+11.44%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...apache_beam/runners/portability/portable_runner.py	`75.60% <0.00%> (-1.72%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.54% <0.00%> (-0.25%)`	⬇️
...am/testing/benchmarks/cloudml/criteo_tft/criteo.py	`0.00% <0.00%> (ø)`
...chmarks/cloudml/cloudml_benchmark_constants_lib.py	`100.00% <0.00%> (ø)`
...m/testing/benchmarks/cloudml/pipelines/workflow.py	`6.94% <0.00%> (ø)`
...ks/python/apache_beam/runners/worker/operations.py	`74.32% <0.00%> (+0.14%)`	⬆️
sdks/python/apache_beam/transforms/environments.py	`88.26% <0.00%> (+0.29%)`	⬆️
...eam/runners/interactive/interactive_environment.py	`92.02% <0.00%> (+0.30%)`	⬆️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`89.24% <0.00%> (+0.48%)`	⬆️
...python/apache_beam/runners/worker/worker_status.py	`75.33% <0.00%> (+0.66%)`	⬆️
... and 160 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

je-ik · 2022-12-22T17:25:53Z

@lukecwik is there any way to get logs of the failed pre commit?
https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/25421/

The tests pass locally, so without the log I'm not sure how to fix the problem. :-/

Fix FlinkRequiresStableInputTest flakiness (apache#21333)

Co-authored-by: Lukasz Cwik <[email protected]>

je-ik · 2023-01-03T13:33:14Z

@lukecwik I finally managed to make the tests pass, so this is ready for final review. I will squash the commits after.

je-ik · 2023-01-23T17:21:25Z

@lukecwik would you have time for review, or could you recommend someone please?

lukecwik

LGTM

lukecwik · 2023-01-23T23:42:36Z

I'm glad that you were able to work through the problems and get the tests fixed and passing.

github-actions bot added flink runners labels Aug 25, 2022

je-ik force-pushed the issue-20812-requires-stable-input-portable branch 2 times, most recently from 0b076df to 54543bb Compare August 25, 2022 13:01

je-ik requested review from ibzib and dmvk August 25, 2022 13:01

je-ik force-pushed the issue-20812-requires-stable-input-portable branch 3 times, most recently from d33b6f1 to 7f54f69 Compare August 25, 2022 13:38

je-ik force-pushed the issue-20812-requires-stable-input-portable branch from 7f54f69 to f7d3e5d Compare August 29, 2022 07:40

je-ik changed the title ~~#20812 handle @RequiresStableInput in portable flink~~ [DO NOT MERGE] #20812 handle @RequiresStableInput in portable flink Aug 30, 2022

je-ik force-pushed the issue-20812-requires-stable-input-portable branch from f7d3e5d to 3e97222 Compare September 5, 2022 09:35

je-ik changed the title ~~[DO NOT MERGE] #20812 handle @RequiresStableInput in portable flink~~ #20812 handle @RequiresStableInput in portable flink Sep 5, 2022

je-ik force-pushed the issue-20812-requires-stable-input-portable branch from 3e97222 to 9445883 Compare September 5, 2022 11:34

je-ik requested a review from lukecwik September 5, 2022 14:19

aaltay requested a review from kileys September 27, 2022 16:02

je-ik force-pushed the issue-20812-requires-stable-input-portable branch 3 times, most recently from 842a84f to d597e4e Compare December 22, 2022 15:14

je-ik force-pushed the issue-20812-requires-stable-input-portable branch from d597e4e to 1d14adf Compare December 23, 2022 10:06

je-ik force-pushed the issue-20812-requires-stable-input-portable branch 2 times, most recently from 80389b4 to 7e12621 Compare January 2, 2023 17:13

je-ik added 2 commits January 2, 2023 18:18

Handle @RequiresStableInput in portable flink (apache#20812)

6b1ee41

[runners-flink] Remove unnnecessary dependency on flink-annotations

a438567

je-ik force-pushed the issue-20812-requires-stable-input-portable branch from 7e12621 to 5919a8d Compare January 2, 2023 17:23

je-ik added 2 commits January 3, 2023 10:11

Fix @RequiresStableInput for portable Flink (apache#20812)

80a0a5e

Fix FlinkRequiresStableInputTest flakiness (apache#21333)

Flink: Tests for stateful stable dofns (apache#20812)

a824fe9

je-ik force-pushed the issue-20812-requires-stable-input-portable branch from 5919a8d to dbd0997 Compare January 3, 2023 09:11

je-ik and others added 3 commits January 3, 2023 13:23

Enable commit for kafka flink portable test

a2fa8d9

Apply suggestions from code review

6335500

Co-authored-by: Lukasz Cwik <[email protected]>

Add callback to BufferingDoFnRunner for flushing SDK harness results

bb622e4

je-ik force-pushed the issue-20812-requires-stable-input-portable branch from dbd0997 to bb622e4 Compare January 3, 2023 12:24

je-ik changed the title ~~[DO NOT MERGE] #20812 handle @RequiresStableInput in portable flink~~ #20812 handle @RequiresStableInput in portable flink Jan 3, 2023

revert changes in website

0694831

github-actions bot removed the website label Jan 3, 2023

je-ik requested a review from lukecwik January 3, 2023 16:54

lukecwik approved these changes Jan 23, 2023

View reviewed changes

lukecwik merged commit b1aba9d into apache:master Jan 23, 2023

je-ik deleted the issue-20812-requires-stable-input-portable branch January 24, 2023 08:34

Abacn mentioned this pull request Apr 18, 2023

Flink testParDoRequiresStableInput flaky #21333

Open

minxhe mentioned this pull request Nov 25, 2024

[cherry-pick] add support for flink 1.18 for Beam Flink Runner linkedin/beam#130

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#20812 handle @RequiresStableInput in portable flink #22889

#20812 handle @RequiresStableInput in portable flink #22889

je-ik commented Aug 25, 2022 •

edited

Loading

je-ik commented Aug 25, 2022

github-actions bot commented Aug 25, 2022

github-actions bot commented Aug 25, 2022

je-ik commented Aug 29, 2022

je-ik commented Aug 30, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 27, 2022

aaltay commented Sep 27, 2022

codecov bot commented Dec 14, 2022 •

edited

Loading

je-ik commented Dec 22, 2022

je-ik commented Jan 3, 2023

je-ik commented Jan 23, 2023

lukecwik left a comment

lukecwik commented Jan 23, 2023

#20812 handle @RequiresStableInput in portable flink #22889

#20812 handle @RequiresStableInput in portable flink #22889

Conversation

je-ik commented Aug 25, 2022 • edited Loading

Fixes #20812 Fixes #21333

GitHub Actions Tests Status (on master branch)

je-ik commented Aug 25, 2022

github-actions bot commented Aug 25, 2022

github-actions bot commented Aug 25, 2022

je-ik commented Aug 29, 2022

je-ik commented Aug 30, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 5, 2022

je-ik commented Sep 27, 2022

aaltay commented Sep 27, 2022

codecov bot commented Dec 14, 2022 • edited Loading

Codecov Report

je-ik commented Dec 22, 2022

je-ik commented Jan 3, 2023

je-ik commented Jan 23, 2023

lukecwik left a comment

Choose a reason for hiding this comment

lukecwik commented Jan 23, 2023

je-ik commented Aug 25, 2022 •

edited

Loading

Fixes #20812
Fixes #21333

codecov bot commented Dec 14, 2022 •

edited

Loading