Creating a Fully Managed Beam Streaming System with Flink Runner on Kubernetes

[talatuyarer]

Hi all,

We created a blog post for Apache beam about how we created a managed service for our Beam jobs on Kubernetes. First part is about General Architecture details. I am also working on challenges and learning on part2. I hope you can publish this blog post on Beam Blog. I believe this will be helpful for another who wants to build similar system.

Thanks

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[liferoad]

This looks great! Thanks for doing this.

cc @damccorm

[talatuyarer]

Thank you @liferoad Do you know how can get approval for this ? Should i ask @damccorm ?
I am working on Part 2 also. It will have more insights. We learned so many things.

[talatuyarer]

assign set of reviewers

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @riteshghorse added as fallback since no labels match configuration

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[liferoad]

Thank you @liferoad Do you know how can get approval for this ? Should i ask @damccorm ? I am working on Part 2 also. It will have more insights. We learned so many things.

Great! Thanks for Part 2!
Let me ask around about who could help reviewing this.

[liferoad]

R: @rszper

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[nika-qubit]

LGTM with a nit.

[damondouglas]

Consider my question non-blocking of the PR. Is the intent for the reader to follow this article to replicate in their own environment?

[talatuyarer]

Consider my question non-blocking of the PR. Is the intent for the reader to follow this article to replicate in their own environment?

Thank you reviewing @damondouglas Our goal is to share experience while building this system. and Show readers it is possible with some few challenges. Whoever wants to build similar system. I believe they can see different issue also too. I can write a tutorial after Part 2 step by step how they can build based on open source examples.

[damondouglas]

This is resolved:

Thank you again for this really helpful blog! Pardon if this comment is already addressed, but I saw what might be the cause of the failing checks. It seems both failed checks point to the one error.

I see: ERROR Author 'talat' is not defined. Please add them to 'website/www/site/data/authors.yml' at https://github.com/apache/beam/actions/runs/6788979982/job/18455154941?pr=29305#step:6:2861

I will check in tomorrow (2023-11-08 PST time zone) to see the status of this PR and review again to assess any resolution of existing comments and pre-commit checks. Thank you for your patience and your contribution. I look forward to trying this out myself.

[talatuyarer]

I believe we cleared all issues but Looks like pictures are very blur. I will change pictures.

[talatuyarer]

@damondouglas Please review it. I updated images and fixed all issues

[damondouglas]

Non-blocking: This image looks a bit small. You can decide to keep it or remove it since you already have the FlinkDeployment manifest as a code snippet. Therefore, my comment is non-blocking.

[damondouglas]

LGTM Thank you so much.