Allow JSON type in TableSchema for BigQuery FILE_LOAD #29923
Conversation
* Warn instead of fail validation and link to documentation * Add test for this use case
|
|
||
| p.run().waitUntilFinish(); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
The write tests also read from bigquery table for validation with pipelines, with exactly same pipelines of the read test. So I have consolidated these two read tests into writeandread.
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
R: @ahmedabu98 |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
Show resolved
Hide resolved
| LOG.warn( | ||
| "Found JSON type in TableSchema for 'FILE_LOADS' write method. \n" | ||
| + "Make sure the TableSchema field is a parsed JSON to ensure the read as a " | ||
| + "JSON type. Otherwise it will read as a raw (escaped) string."); |
There was a problem hiding this comment.
Does this mean the TableRow value should be a parsed JSON?
There was a problem hiding this comment.
Yes. This is documented behavior of BQ FILE_LOAD API
e.g., instead of "someColumn": "[1]", it needs to be "someColumn": [1]. Otherwise it will write as a single string (string is also valid json object)
There was a problem hiding this comment.
I see, can we change it to Make sure the TableRow value is a parsed JSON...?
Can we also include a link to the documented behavior?
There was a problem hiding this comment.
ah I see. I meant to TableRow value. It was a typo.
| @@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped( | |||
| !getPropagateSuccessfulStorageApiWrites(), | |||
| "withPropagateSuccessfulStorageApiWrites only supported when using storage api writes."); | |||
|
|
|||
| // Batch load jobs currently support JSON data insertion only with CSV files | |||
| // Beam does not yet support Batch load jobs with Avro files | |||
There was a problem hiding this comment.
Remove this comment? I think it's supported for all types now
There was a problem hiding this comment.
thanks for catching this
|
R: @ahmedabu98 missed tracking this, PTAL |
|
Failed test FhirIOWriteIT known flaky, not related to the change, merging for now |
Fixes #23025 ; fixes #27448
Warn instead of fail validation and link to documentation
Add test for this use case
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.