Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BigQuery StorageWrite: JsonToProtoMessage numeric error #1516

Closed
caseyduquettesc opened this issue Feb 7, 2022 · 1 comment · Fixed by #1596, #1648 or #1644
Closed

BigQuery StorageWrite: JsonToProtoMessage numeric error #1516

caseyduquettesc opened this issue Feb 7, 2022 · 1 comment · Fixed by #1596, #1648 or #1644
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@caseyduquettesc
Copy link

Environment details

  1. BigQuery StorageWrite
  2. OS type and version: MacOS 11.6.3
  3. Java version: 11
  4. version(s): 2.8.4

Steps to reproduce

  1. Create a bigquery table with only one field of type NUMERIC and with name "d"
  2. Use JsonStreamWriter with this JSON Array [{"d":1}]
  3. run

Stack trace

java.lang.IllegalArgumentException: JSONObject does not have a bytes field at root.d.
	at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.fillField(JsonToProtoMessage.java:351)
	at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessageImpl(JsonToProtoMessage.java:176)
	at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessage(JsonToProtoMessage.java:115)
	at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:140)
	at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:99)
	at persistence.BigqueryStreamingClient.streamJsonArray(BigqueryStreamingClient.java:80)

External references such as API reference guides

  • ?

Any additional information below

It seems if you try to write an int/long to a NUMERIC field, it fails because the proto converter requires a bytes object for NUMERIC's. Looking at

java-bigquerystorage/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java

Line 229 in 4c532b0

case BYTES:
it looks like only Strings or ByteStrings are converted to bytes. The end result is not being able to write numeric java types to the NUMERIC BQ column.
@product-auto-label product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Feb 7, 2022
@stephaniewang526 stephaniewang526 self-assigned this Feb 7, 2022
@stephaniewang526 stephaniewang526 added type: question Request for information or clarification. Not an issue. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed type: question Request for information or clarification. Not an issue. labels Feb 7, 2022
@dark0dave dark0dave mentioned this issue Mar 30, 2022
Merged
@stephaniewang526 stephaniewang526 removed their assignment Apr 28, 2022
@dark0dave dark0dave mentioned this issue May 7, 2022
Merged
gcf-merge-on-green bot pushed a commit that referenced this issue May 13, 2022
@dark0dave
fix: ints/longs are numerics (#1596)
d046c8d 
Should allow users to pass ints and longs as numerics.

Fixes #1516
@release-please release-please bot mentioned this issue May 13, 2022
Merged
gcf-merge-on-green bot pushed a commit that referenced this issue May 24, 2022
@release-please
chore(main): release 2.14.0 (#1644)
da4e7fc 
🤖 I have created a release *beep* *boop*
---


## [2.14.0](v2.13.0...v2.14.0) (2022-05-19)


### Features

* add build scripts for native image testing in Java 17 ([#1440](#1440)) ([#1655](#1655)) ([ac2dfaf](ac2dfaf))


### Bug Fixes

* Add a throwException behavior when the StreamWriter inflight queue is full ([#1642](#1642)) ([4dcf0d5](4dcf0d5))
* add extra JsonWriterTest to show that the LimitBehavior addition is not breaking ([#1643](#1643)) ([320f5fc](320f5fc))
* ints/longs are numerics ([#1596](#1596)) ([d046c8d](d046c8d)), closes [#1516](#1516)


### Dependencies

* update arrow.version to v8 ([#1645](#1645)) ([06e3c34](06e3c34))
* update dependency com.google.cloud:google-cloud-shared-dependencies to v2.12.0 ([#1654](#1654)) ([ec4f60b](ec4f60b))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@feamcor
Copy link

feamcor commented Jun 15, 2022

Please see #1648 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Milestone
No milestone
3 participants
@feamcor @stephaniewang526 @caseyduquettesc