Improve performance of BigQueryIO connector when withPropagateSuccessfulStorageApiWrites(true) is used #31840
Conversation
…Write API proto's to TableRows when withPropagateSuccessfulStorageApiWrites(true) is used.
|
Assigning reviewers. If you would like to opt out of this review, comment R: @damondouglas for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
@ahmedabu98 - I don't have permissions to merge the PR. Could you please, or someone with committer privileges, merge it and mention which release this PR will be available in? |
|
Release 2.58.0 has been cut a while ago. This will make it in for version 2.59.0. Sorry for not asking earlier, but could you add an entry in CHANGES.md mentioning this improvement? |
|
@ahmedabu98 - added a line to the 2.59.0 section. When is the SNAPSHOT build going to be available and what's the tentative release date? |
|
We build snapshots on a daily basis, so probably tomorrow. If all goes well, 2.59.0 should be released mid/late September |
…fulStorageApiWrites(true) is used (apache#31840) * Performance improvements related to conversion of BigQuery's Storage Write API proto's to TableRows when withPropagateSuccessfulStorageApiWrites(true) is used. * Fix spotless findings. * Update CHANGES.md * Update CHANGES.md - moved the entry to 2.59.0 section.
Minor updates to the way TableRows are re-constructed from the proto messages used for calls to the Storage Write API.
Profiling showed over 50% improvement in the CPU utilization in the code branch related to this re-construction.
Additional optimization is possible if the STORAGE_API_AT_LEAST_ONCE method is used. Another PR will be submitted as the follow up to this one.