Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Support for Apache Flink 1.18 in Beam Runners #30789

Closed
1 of 16 tasks
chitluri opened this issue Mar 28, 2024 · 15 comments
Closed
1 of 16 tasks

[Feature Request]: Support for Apache Flink 1.18 in Beam Runners #30789

chitluri opened this issue Mar 28, 2024 · 15 comments

Comments

@chitluri
Copy link

chitluri commented Mar 28, 2024

What would you like to happen?

[Flink Runner] Question

Description
Our organization is in the process of updating our Flink clusters to the latest version, Flink 1.18. We currently use beam-runners-flink-1.16 to run Apache Beam pipelines on our Flink 1.16 cluster. We are concerned that our Beam pipelines will fail after the upgrade due to incompatibility with Flink 1.18. This is based on the information provided at https://beam.apache.org/documentation/runners/flink/: Flink Version Compatibility section.

Ask
We would like to inquire about the timeline and plans for supporting Apache Flink 1.18 in Apache Beam. Is there an ongoing effort to add compatibility for Flink 1.18, or is it on the roadmap for a future release?

Additional Context

  • Our pipelines are critical to our business operations, and we need to ensure a smooth transition during the Flink upgrade.
  • We are willing to assist or contribute to the effort of adding Flink 1.18 support if guidance is provided on what is needed.

Potential Impact

  • Without support for Flink 1.18, we may face significant challenges in maintaining our data processing workflows.
  • Early information on support timelines will help us plan our upgrade strategy and resource allocation more effectively.

Questions

  1. Is there a planned release date for a version of Beam that supports Flink 1.18?
  2. Are there any known workarounds or interim solutions for running Beam pipelines on Flink 1.18?
  3. How can the community contribute to accelerating support for Flink 1.18 in Beam?

We appreciate any information or guidance the Apache Beam community can provide on this matter.

Thank you for your time and support.

eBay

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@thebozzcl
Copy link
Contributor

+1 to this issue, my org is in the same boat (long story short: company-wide migration to Java 17, we need Flink 1.18, but we can't use it because of Beam's lack of support).

Some fun extra context: #30197

This MR to support Flink 1.17 is currently blocked because the upgrade would break compatibility with checkpoints created in older versions.

@efranc118
Copy link

+1 to this issue, we are also in the similar situation at my org as well.

@mmk4096
Copy link

mmk4096 commented Mar 29, 2024

+1 also interested.

@twosom
Copy link
Contributor

twosom commented Mar 31, 2024

+1 for this

@danielmfinn
Copy link

+1. Would help support of Java 17.

@aniinbkln
Copy link

+1 for this

1 similar comment
@BohdanBilonoh
Copy link

+1 for this

@ConfuzedCoder
Copy link

  • for this

@m0hit-aggarwal
Copy link

+1 for this.

@Abacn
Copy link
Contributor

Abacn commented Apr 4, 2024

Hi all, looks like this is highly wanted. Could Flink runner user help testing #30197 for their use cases?

@Abacn
Copy link
Contributor

Abacn commented Apr 4, 2024

As of timeline, as an open source project the Flink runner component has been community driven. Any contribution is welcome.

@thebozzcl
Copy link
Contributor

I forked from that PR to add support for 1.18. It's a trivial change, too: thebozzcl@432643b

So far this is working fine in our tests, with a caveat - in its current status, there's some incompatibilities with the publicly released libraries (mostly around SerializablePipelineOptions). For our tests, I had to use custom-built versions of a few libraries based on the repo:

  • beam-runners-core-java
  • beam-runners-java-fn-execution
  • beam-runners-java-job-service
  • beam-sdks-java-core

I'm hopeful that once the PR for 1.17 is merged, adding support for 1.18 will be trivial. We're going to try using these libraries in one of our more complex pipelines, see if everything still works.

@thebozzcl
Copy link
Contributor

Also, please note that the codebase in #30197 has diverged from the main branch quite a bit, so I'm not even sure my changes will apply once that's rebased.

@dan-clst
Copy link

@thebozzcl looks like 1.17 support was just merged in #30197

thebozzcl added a commit to thebozzcl/beam that referenced this issue Apr 19, 2024
@je-ik je-ik assigned je-ik and unassigned je-ik Apr 26, 2024
je-ik pushed a commit that referenced this issue May 7, 2024
@je-ik
Copy link
Contributor

je-ik commented May 7, 2024

Closed via #31062

@je-ik je-ik closed this as completed May 7, 2024
@github-actions github-actions bot added this to the 2.57.0 Release milestone May 7, 2024
je-ik added a commit to je-ik/beam that referenced this issue May 13, 2024
Abacn added a commit that referenced this issue May 13, 2024
je-ik added a commit to je-ik/beam that referenced this issue May 23, 2024
minxhe pushed a commit to minxhe/beam that referenced this issue Nov 21, 2024
minxhe pushed a commit to minxhe/beam that referenced this issue Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests