Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Python's Storage API streaming exactly-once writes on DataflowV2 is broken with autosharding, and it's the only option available #28587

Closed
3 of 15 tasks
ahmedabu98 opened this issue Sep 21, 2023 · 0 comments · Fixed by #28592
Closed
3 of 15 tasks

[Bug]: Python's Storage API streaming exactly-once writes on DataflowV2 is broken with autosharding, and it's the only option available #28587

ahmedabu98 opened this issue Sep 21, 2023 · 0 comments · Fixed by #28592
Assignees
@ahmedabu98
Labels
bug io java P1 python
Milestone
2.52.0 Release

Comments

@ahmedabu98
Copy link
Contributor

ahmedabu98 commented Sep 21, 2023 •
edited
Loading

What happened?

Java's Storage Write API streaming writes is broken on Dataflow runner V2 when autosharding is enabled. The Python wrapper uses the Java implementation and runs exclusively on Runner V2.

To provide a workaround and unblock users, we should enable setting a fixed number of shards.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@ahmedabu98 ahmedabu98 self-assigned this Sep 21, 2023
@ahmedabu98 ahmedabu98 added this to the 2.51.0 Release milestone Sep 21, 2023
@ahmedabu98 ahmedabu98 changed the title [Bug]: Python's Storage API streaming writes via xlang is broken on Dataflow because it only runs on Runner V2 [Bug]: Python's Storage API streaming exactly-once writes on DataflowV2 is broken with autosharding, and it's the only option available Sep 21, 2023
@ahmedabu98 ahmedabu98 mentioned this issue Sep 21, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ahmedabu98 ahmedabu98

Labels
bug io java P1 python
Projects
None yet
Milestone
2.52.0 Release
Development

Successfully merging a pull request may close this issue.

[Python BQ] Allow setting a fixed number of Storage API streams ahmedabu98/beam
1 participant
@ahmedabu98