Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flink versioning issues #31168

Merged
merged 6 commits into from
May 3, 2024
Merged

Fix flink versioning issues #31168

merged 6 commits into from
May 3, 2024

Conversation

damccorm
Copy link
Contributor

@damccorm damccorm commented May 3, 2024

As I was doing some work to update Dataflow's notebooks product, I noticed that interactive beam on flink doesn't work with 2.56. When I dug deeper, I found a bunch of references that weren't updated in #30197

This tries to cover them all, and adds the known issue.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@damccorm
Copy link
Contributor Author

damccorm commented May 3, 2024

R: @je-ik @Abacn

Copy link
Contributor

github-actions bot commented May 3, 2024

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@@ -14,7 +14,7 @@
},
"JavaTestProperties": {
"SUPPORTED_VERSIONS": ["8", "11", "17"],
"FLINK_VERSIONS": ["1.13", "1.14", "1.15"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a side note

this is stale implies this field is not used, actually currently this action is only used in update_python_dependencies and publish_github_release_notes workflow.

Currently there is hard coded python/java versions everywhere in our GHAs. As futurre clean up we could make use of this action to have single source of version setting for actions

@@ -1591,7 +1591,7 @@ def _add_argparse_args(cls, parser):
class FlinkRunnerOptions(PipelineOptions):

# These should stay in sync with gradle.properties.
PUBLISHED_FLINK_VERSIONS = ['1.12', '1.13', '1.14', '1.15', '1.16']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note - this would essentially prevent python user using flink 1.17 for Beam 2.56.0, and not prevent user choosing 1.12,1.13. Consider update the known issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I sorta mentioned this, but I updated to be clearer/more explicit. Thanks

Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@damccorm
Copy link
Contributor Author

damccorm commented May 3, 2024

The test failures all seem to be preexisting, but I don't feel good about merging this until we get a greener signal since flakes are blocking our test coverage.

#31169 should fix it, at which point I'll merge in master and try this again

@damccorm
Copy link
Contributor Author

damccorm commented May 3, 2024

Typescript tests are failing, but they're also permared; no meaningful signal there

@je-ik
Copy link
Contributor

je-ik commented May 3, 2024

Looks like we need some post-commit check to cover this?

@damccorm
Copy link
Contributor Author

damccorm commented May 3, 2024

Looks like we need some post-commit check to cover this?

Probably - I'm surprised nothing failed here

@damccorm damccorm merged commit 7090260 into master May 3, 2024
83 of 85 checks passed
@damccorm damccorm deleted the users/damccorm/ibFlink branch May 3, 2024 17:04
@je-ik
Copy link
Contributor

je-ik commented May 7, 2024

Do we have a github issue for fixing the known issue? I'm struggling to find it. The mentioned #31168 links to this PR.

@damccorm
Copy link
Contributor Author

damccorm commented May 7, 2024

No, I just went straight to PR since it was easier than enumerating the problems in an issue first

@je-ik
Copy link
Contributor

je-ik commented May 7, 2024

Ah, sorry, I did not mean issue for this PR, I meant tracking issue for the mentioned problem https://github.com/apache/beam/pull/31168/files#diff-d975bf659606195d2165918f93e1cf680ef68ea3c9cab994f033705fea8238b2R131

@je-ik
Copy link
Contributor

je-ik commented May 7, 2024

Context: there is already PR for 1.18, it would be good to avoid similar issue in the future, yet I'm not sure I understand the problem correctly.

@damccorm
Copy link
Contributor Author

damccorm commented May 7, 2024

Basically, the problem is that we track Flink version support in a bunch of places which need updated when we add a version (and more importantly when we remove a version). The most important ones I noticed are:

  1. PUBLISHED_FLINK_VERSIONS = ['1.14', '1.15', '1.16', '1.17']
    - controls which flink versions can be used from Python
  2. - controls which flink version is used by interactive beam

But really all the files in this PR need updated anytime we update flink version availability

@je-ik
Copy link
Contributor

je-ik commented May 8, 2024

Understood. I was confused by what else needs to be done, but the problem is that this PR was merged after the RC of 2.56.0 was built, so this PR fixes it for 2.57.0.

Got it, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants