Support dynamic destinations with Python Storage API #30045
Conversation
|
Assigning reviewers. If you would like to opt out of this review, comment R: @AnandInguva for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
R: @chamikaramj |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
| input_beam_rows = ( | ||
| input_rows | ||
| | "Wrap in Beam Row" >> | ||
| beam.Map(lambda row: beam.Row(destination=row[0], record=row[ |
There was a problem hiding this comment.
Is it possible for row[0] or row[1] to be None?
There was a problem hiding this comment.
Only if the dynamic destination function returns None, which would be weird from the user's side.
|
GCP Xlang Postcommit Dataflow run: https://github.com/apache/beam/actions/runs/7603738594/job/20705831041?pr=30045 |
|
LGTM. |
Python's write to BQ Storage API connector uses the cross-language framework. This PR adds support for dynamic destinations to this write mode.
Destination for each row is determined in the Python side using the callable. Then we send each row with its destination over to Java. We send Java a magic string to communicate that dynamic destinations are being used so that it handles this correctly.