Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python BQ] Follow up of #28592: Allow setting a fixed number of Storage API streams #28618

Merged
merged 10 commits into from
Sep 22, 2023

Conversation

ahmedabu98
Copy link
Contributor

Had to fix some lint checks in #28592 which was reverted in #28613

@ahmedabu98
Copy link
Contributor Author

Run Python_Xlang_Gcp_Dataflow PostCommit

@ahmedabu98
Copy link
Contributor Author

Run Python_Xlang_Gcp_Direct PostCommit

@codecov
Copy link

codecov bot commented Sep 22, 2023

Codecov Report

Merging #28618 (987a8df) into master (426dbd3) will increase coverage by 0.01%.
Report is 2 commits behind head on master.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master   #28618      +/-   ##
==========================================
+ Coverage   72.20%   72.22%   +0.01%     
==========================================
  Files         684      684              
  Lines      101131   101133       +2     
==========================================
+ Hits        73023    73042      +19     
+ Misses      26529    26512      -17     
  Partials     1579     1579              
Flag Coverage Δ
python 82.78% <50.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
sdks/python/apache_beam/io/gcp/bigquery.py 69.92% <50.00%> (-0.05%) ⬇️

... and 6 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@ahmedabu98
Copy link
Contributor Author

retest this please

@ahmedabu98
Copy link
Contributor Author

Run Python_Xlang_Gcp_Direct PostCommit

@ahmedabu98
Copy link
Contributor Author

Run Python_Xlang_Gcp_Dataflow PostCommit

@ahmedabu98 ahmedabu98 marked this pull request as ready for review September 22, 2023 18:33
@github-actions
Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@@ -176,6 +176,13 @@ public void validate() {
!Strings.isNullOrEmpty(this.getErrorHandling().getOutput()),
invalidConfigMessage + "Output must not be empty if error handling specified.");
}

if (this.getAutoSharding() != null && this.getAutoSharding()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this be a check on the main BigQuery transform?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A similar check does exist in the top-level BigQueryIO transform:

if (getAutoSharding()) {
if (method == Method.STORAGE_WRITE_API && getStorageApiNumStreams(bqOptions) > 0) {
LOG.warn(
"Both numStorageWriteApiStreams and auto-sharding options are set. Will default to auto-sharding."
+ " To set a fixed number of streams, do not enable auto-sharding.");

...along with many other warnings we put in place just because we don't want to push breaking changes. In the case of a Beam 3 type of event, we would make all these checks throw errors. With that in mind, I added a checkArgument when introducing the numStreams option.

@kennknowles
Copy link
Member

Also: Please don't merge with failing GitHub statuses. Kick them to green. And the coverage complaint seems legitimate.

@ahmedabu98
Copy link
Contributor Author

@kennknowles some FhirIO tests are very flaky (maybe perma-failing) on Github workflow #28635

kennknowles pushed a commit that referenced this pull request Sep 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants