feat: allow java client to handle schema change during same stream name #1964

GaoleMeng · 2023-01-27T21:59:39Z

We will consider multiplexing to be entered during schema switch.

Also fixed the clirrr error

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

prerequisite for multiplexing client

new stream name as a switch of destinationt

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…nt or not

possible the proto schema does not contain this field

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

case

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

yirutang · 2023-01-27T22:01:57Z

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java

            && !streamName.isEmpty()
-            && !originalRequest.getWriteStream().equals(streamName)) {
+            && !originalRequest.getWriteStream().equals(streamName))
+            || !originalRequest.getProtoRows().getWriterSchema().equals(writerSchema)) {


could this be null?

No as we will always pass in write schema on the original request, but let's check that to make sure

yirutang · 2023-01-27T22:05:56Z

...bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerTest.java

@@ -247,10 +247,10 @@ public void testAppendInSameStream_switchSchema() throws Exception {
        // We will get the request as the pattern of:
        // (writer_stream: t1, schema: schema1)
        // (writer_stream: _, schema: _)
-        // (writer_stream: _, schema: schema3)


I am confused, so previously, just switch the schema works?

previously the schema is passed to backend, but since multiplexing is not entered, the stream name is not enforeced and backend can't use the new schema to parse the value

…tream name

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…me (googleapis#1964) * feat: Split writer into connection worker and wrapper, this is a prerequisite for multiplexing client * feat: add connection worker pool skeleton, used for multiplexing client * feat: add Load api for connection worker for multiplexing client * feat: add multiplexing support to connection worker. We will treat every new stream name as a switch of destinationt * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: port the multiplexing client core algorithm and basic tests also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: wire multiplexing connection pool to stream writer * feat: some fixes for multiplexing client * feat: fix some todos, and reject the mixed behavior of passed in client or not * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * feat: add getInflightWaitSeconds implementation * feat: Add schema comparision in connection loop to ensure schema update for the same stream name can be notified * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add schema update support to multiplexing * fix: fix windows build bug: windows Instant resolution is different with linux * fix: fix another failing tests for windows build * fix: fix another test failure for Windows build * feat: Change new thread for each retry to be a thread pool to avoid create/tear down too much threads if lots of retries happens * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add back the background executor provider that's accidentally removed * feat: throw error when use connection pool for explicit stream * fix: Add precision truncation to the passed in value from JSON float and double type. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * modify the bom version * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix deadlockissue in ConnectionWorkerPool * fix: fix deadlock issue during close + append for multiplexing * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: fix one potential root cause of deadlock issue for non-multiplexing case * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add timeout to inflight queue waiting, and also add some extra log * feat: allow java client lib handle switch table schema for the same stream name * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

* fix: at connection level, retry for internal errors * . * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * . * . * . * . * . * feat: allow java client to handle schema change during same stream name (#1964) * feat: Split writer into connection worker and wrapper, this is a prerequisite for multiplexing client * feat: add connection worker pool skeleton, used for multiplexing client * feat: add Load api for connection worker for multiplexing client * feat: add multiplexing support to connection worker. We will treat every new stream name as a switch of destinationt * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: port the multiplexing client core algorithm and basic tests also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: wire multiplexing connection pool to stream writer * feat: some fixes for multiplexing client * feat: fix some todos, and reject the mixed behavior of passed in client or not * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * feat: add getInflightWaitSeconds implementation * feat: Add schema comparision in connection loop to ensure schema update for the same stream name can be notified * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add schema update support to multiplexing * fix: fix windows build bug: windows Instant resolution is different with linux * fix: fix another failing tests for windows build * fix: fix another test failure for Windows build * feat: Change new thread for each retry to be a thread pool to avoid create/tear down too much threads if lots of retries happens * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add back the background executor provider that's accidentally removed * feat: throw error when use connection pool for explicit stream * fix: Add precision truncation to the passed in value from JSON float and double type. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * modify the bom version * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix deadlockissue in ConnectionWorkerPool * fix: fix deadlock issue during close + append for multiplexing * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: fix one potential root cause of deadlock issue for non-multiplexing case * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add timeout to inflight queue waiting, and also add some extra log * feat: allow java client lib handle switch table schema for the same stream name * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * fix: remove unrecoverable connection from connection pool during multiplexing (#1967) * feat: Split writer into connection worker and wrapper, this is a prerequisite for multiplexing client * feat: add connection worker pool skeleton, used for multiplexing client * feat: add Load api for connection worker for multiplexing client * feat: add multiplexing support to connection worker. We will treat every new stream name as a switch of destinationt * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: port the multiplexing client core algorithm and basic tests also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: wire multiplexing connection pool to stream writer * feat: some fixes for multiplexing client * feat: fix some todos, and reject the mixed behavior of passed in client or not * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * feat: add getInflightWaitSeconds implementation * feat: Add schema comparision in connection loop to ensure schema update for the same stream name can be notified * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add schema update support to multiplexing * fix: fix windows build bug: windows Instant resolution is different with linux * fix: fix another failing tests for windows build * fix: fix another test failure for Windows build * feat: Change new thread for each retry to be a thread pool to avoid create/tear down too much threads if lots of retries happens * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add back the background executor provider that's accidentally removed * feat: throw error when use connection pool for explicit stream * fix: Add precision truncation to the passed in value from JSON float and double type. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * modify the bom version * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix deadlockissue in ConnectionWorkerPool * fix: fix deadlock issue during close + append for multiplexing * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: fix one potential root cause of deadlock issue for non-multiplexing case * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add timeout to inflight queue waiting, and also add some extra log * feat: allow java client lib handle switch table schema for the same stream name * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Gaole Meng <[email protected]>

* chore: update dependency versions in java templates * update other templates Source-Link: googleapis/synthtool@0b86c72 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-java:latest@sha256:68ba5f5164a4b55529d358bb262feaa000536a0c62980727dd05a91bbb47ea5e

* chore: update dependency versions in java templates * update other templates Source-Link: googleapis/synthtool@0b86c72 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-java:latest@sha256:68ba5f5164a4b55529d358bb262feaa000536a0c62980727dd05a91bbb47ea5e Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

GaoleMeng and others added 30 commits September 13, 2022 01:58

feat: Split writer into connection worker and wrapper, this is a

5a63d95

prerequisite for multiplexing client

feat: add connection worker pool skeleton, used for multiplexing client

5a13302

Merge branch 'main' into main

0297204

feat: add Load api for connection worker for multiplexing client

8a81ad3

Merge remote-tracking branch 'upstream/main'

68fd040

Merge remote-tracking branch 'upstream/main'

3106dae

Merge branch 'googleapis:main' into main

5bf04e5

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

2fc7551

feat: add multiplexing support to connection worker. We will treat every

7a6d919

new stream name as a switch of destinationt

🦉 Updates from OwlBot post-processor

3ba7659

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Updates from OwlBot post-processor

f379a78

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9307776

🦉 Updates from OwlBot post-processor

de73013

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

feat: port the multiplexing client core algorithm and basic tests

19005a1

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

Merge branch 'googleapis:main' into main

c5d14ba

🦉 Updates from OwlBot post-processor

644360a

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

3099d82

Merge branch 'googleapis:main' into main

e707dd6

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9e7a8fa

Merge branch 'googleapis:main' into main

31f1755

feat: wire multiplexing connection pool to stream writer

44c36fc

feat: some fixes for multiplexing client

87a4036

Merge remote-tracking branch 'upstream/main'

c92ea1b

Merge branch 'googleapis:main' into main

019520c

feat: fix some todos, and reject the mixed behavior of passed in clie…

47893df

…nt or not

Merge remote-tracking branch 'upstream/main'

8bd4e6a

Merge remote-tracking branch 'upstream/main'

83409b0

Merge branch 'googleapis:main' into main

f7dd72d

Merge branch 'googleapis:main' into main

a48399f

feat: fix the bug that we may peek into the write_stream field but it's

6789bc9

possible the proto schema does not contain this field

gcf-owl-bot bot and others added 10 commits January 20, 2023 00:07

🦉 Updates from OwlBot post-processor

6a512e8

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

05edc2f

Merge branch 'googleapis:main' into main

7d3da74

Merge branch 'googleapis:main' into main

ecf6807

Merge branch 'googleapis:main' into main

057dab9

fix: fix one potential root cause of deadlock issue for non-multiplexing

5db46a2

case

🦉 Updates from OwlBot post-processor

32e9d33

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

183941a

Add timeout to inflight queue waiting, and also add some extra log

f93f89e

Merge branch 'googleapis:main' into main

e2ec7e7

GaoleMeng requested review from a team and rosiezou January 27, 2023 21:59

product-auto-label bot added size: m Pull request size is medium. api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. labels Jan 27, 2023

GaoleMeng requested a review from yirutang January 27, 2023 22:00

yirutang reviewed Jan 27, 2023

View reviewed changes

feat: allow java client lib handle switch table schema for the same s…

1be6ab4

…tream name

GaoleMeng requested a review from a team as a code owner January 31, 2023 04:27

yirutang approved these changes Jan 31, 2023

View reviewed changes

GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 31, 2023

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 31, 2023

🦉 Updates from OwlBot post-processor

544f063

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

GaoleMeng merged commit 305f71e into googleapis:main Jan 31, 2023

release-please bot mentioned this pull request Feb 1, 2023

chore(main): release 2.29.0 #1969

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: allow java client to handle schema change during same stream name #1964

feat: allow java client to handle schema change during same stream name #1964

GaoleMeng commented Jan 27, 2023 •

edited

Loading

yirutang Jan 27, 2023

GaoleMeng Jan 31, 2023

yirutang Jan 27, 2023

GaoleMeng Jan 31, 2023

feat: allow java client to handle schema change during same stream name #1964

feat: allow java client to handle schema change during same stream name #1964

Conversation

GaoleMeng commented Jan 27, 2023 • edited Loading

yirutang Jan 27, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 31, 2023

Choose a reason for hiding this comment

yirutang Jan 27, 2023

Choose a reason for hiding this comment

GaoleMeng Jan 31, 2023

Choose a reason for hiding this comment

GaoleMeng commented Jan 27, 2023 •

edited

Loading