feat: add update schema support for multiplexing #1867

GaoleMeng · 2022-11-05T02:16:35Z

To make this happen, we will store a mapping from stream name to updated schema mapping inside connection worker pool. Whenever the json writer accept one append, we will check the cache to see whether there is one updated schema and compared with the current one. Then recreate the stream writer if there is different schema

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

prerequisite for multiplexing client

new stream name as a switch of destinationt

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…nt or not

possible the proto schema does not contain this field

…te for the same stream name can be notified

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

yirutang · 2022-11-07T21:43:16Z

...bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerPool.java

+
+    @Override
+    public void onSuccess(AppendRowsResponse response) {
+      streamNameToUpdatedSchema.put(streamName, response.getUpdatedSchema());


After you refreshWriter, you need to mark the entry here as null. I think it is better we keep a schema on the Writer level.

The problem is there could be multiple stream writers that use the same stream name that all need refreshWriter to be triggered whenever there is a updated schema

So we can't directly nullify the updated schema for a given stream name, otherwise some streamwriter might not be able to get the updated schema correctly.

As discussed offline, changed to use timestamp pattern

If the timestamp used in the current stream writer is older than the updated schema version,change to use updated schema

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java

yirutang · 2022-11-09T21:16:45Z

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java

@@ -398,7 +397,7 @@ public static StreamWriter.Builder newBuilder(String streamName) {

  /** Thread-safe getter of updated TableSchema */
  public synchronized TableSchema getUpdatedSchema() {
-    return singleConnectionOrConnectionPool.getUpdatedSchema();
+    return singleConnectionOrConnectionPool.getUpdatedSchema(this);


There is a little meaning change in this field now. Previously, it will only return an Updated Schema when a schema update happens during the life time of this StreamWriter. Now it will always return the "current schema" of our knowledge. May worth explain this a bit since Dataflow is going to use this field.

Added one line comment

yirutang · 2022-11-09T21:33:05Z

...bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorkerPool.java

+  /*
+   * Contains the mapping from stream name to updated schema.
+   */
+  private Map<String, TableSchema> streamNameToUpdatedSchema = new ConcurrentHashMap<>();


Is there a way to only cache to the level of table, the size of this map could be huge, if it is per stream one table schema.

Changed to cache table name

yirutang · 2022-11-10T22:24:52Z

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java

@@ -720,7 +722,7 @@ private AppendRequestAndResponse pollInflightRequestQueue() {
  }

  /** Thread-safe getter of updated TableSchema */
-  public synchronized TableSchema getUpdatedSchema() {
+  public synchronized TableSchemaAndTimestamp getUpdatedSchema() {


This is breaking change. Lets just add a new method instead of change this old method.

This method should not been used as public,
let's fallback to package private

Yeah, I agree that for ConnectionWorker this method shouldn't be public at all. But it seems the method on StreamWriter also changed?

yirutang · 2022-11-10T23:21:14Z

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/StreamWriter.java

@@ -147,11 +152,11 @@ long getInflightWaitSeconds(StreamWriter streamWriter) {
      return connectionWorker().getInflightWaitSeconds();
    }

-    TableSchema getUpdatedSchema() {
+    TableSchemaAndTimestamp getUpdatedSchema(StreamWriter streamWriter) {


Sorry, I mean this is actually a breaking change? Dataflow will use this method.

Discussed offline, let's used timestamp on streamwriter when returning schema

yirutang · 2022-11-10T23:22:11Z

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/ConnectionWorker.java

@@ -720,7 +722,7 @@ private AppendRequestAndResponse pollInflightRequestQueue() {
  }

  /** Thread-safe getter of updated TableSchema */
-  public synchronized TableSchema getUpdatedSchema() {
+  public synchronized TableSchemaAndTimestamp getUpdatedSchema() {


Yeah, I agree that for ConnectionWorker this method shouldn't be public at all. But it seems the method on StreamWriter also changed?

yirutang · 2022-11-11T22:36:55Z

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java

-        refreshWriter(this.streamWriter.getUpdatedSchema());
+      TableSchema updatedSchemaAndTime = this.streamWriter.getUpdatedSchema();
+      // Create a new stream writer internally if a new updated schema is reported from backend.
+      if (updatedSchemaAndTime != null && !this.tableSchema.equals(updatedSchemaAndTime)) {


We can directly use streamWriter.getUpdatedSchema() != null?

GaoleMeng and others added 30 commits September 13, 2022 01:58

feat: Split writer into connection worker and wrapper, this is a

5a63d95

prerequisite for multiplexing client

feat: add connection worker pool skeleton, used for multiplexing client

5a13302

Merge branch 'main' into main

0297204

feat: add Load api for connection worker for multiplexing client

8a81ad3

Merge remote-tracking branch 'upstream/main'

68fd040

Merge remote-tracking branch 'upstream/main'

3106dae

Merge branch 'googleapis:main' into main

5bf04e5

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

2fc7551

feat: add multiplexing support to connection worker. We will treat every

7a6d919

new stream name as a switch of destinationt

🦉 Updates from OwlBot post-processor

3ba7659

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Updates from OwlBot post-processor

f379a78

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9307776

🦉 Updates from OwlBot post-processor

de73013

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

feat: port the multiplexing client core algorithm and basic tests

19005a1

also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset

Merge branch 'googleapis:main' into main

c5d14ba

🦉 Updates from OwlBot post-processor

644360a

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

3099d82

Merge branch 'googleapis:main' into main

e707dd6

Merge branch 'main' of https://github.com/GaoleMeng/java-bigquerystorage

9e7a8fa

Merge branch 'googleapis:main' into main

31f1755

feat: wire multiplexing connection pool to stream writer

44c36fc

feat: some fixes for multiplexing client

87a4036

Merge remote-tracking branch 'upstream/main'

c92ea1b

Merge branch 'googleapis:main' into main

019520c

feat: fix some todos, and reject the mixed behavior of passed in clie…

47893df

…nt or not

Merge remote-tracking branch 'upstream/main'

8bd4e6a

Merge remote-tracking branch 'upstream/main'

83409b0

Merge branch 'googleapis:main' into main

f7dd72d

Merge branch 'googleapis:main' into main

a48399f

feat: fix the bug that we may peek into the write_stream field but it's

6789bc9

possible the proto schema does not contain this field

GaoleMeng and others added 3 commits November 3, 2022 16:51

feat: Add schema comparision in connection loop to ensure schema upda…

d1b7740

…te for the same stream name can be notified

🦉 Updates from OwlBot post-processor

e4cd529

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'googleapis:main' into main

74ff1c4

GaoleMeng requested review from a team and aribray November 5, 2022 02:16

product-auto-label bot added size: l Pull request size is large. api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. labels Nov 5, 2022

GaoleMeng requested a review from yirutang November 7, 2022 21:17

yirutang reviewed Nov 7, 2022

View reviewed changes

yirutang reviewed Nov 9, 2022

View reviewed changes

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java Outdated Show resolved Hide resolved

...oud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonStreamWriter.java Outdated Show resolved Hide resolved

yirutang reviewed Nov 9, 2022

View reviewed changes

yirutang approved these changes Nov 10, 2022

View reviewed changes

yirutang reviewed Nov 10, 2022

View reviewed changes

GaoleMeng requested a review from a team as a code owner November 10, 2022 23:14

yirutang reviewed Nov 10, 2022

View reviewed changes

GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 11, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 11, 2022

yirutang reviewed Nov 11, 2022

View reviewed changes

yirutang approved these changes Nov 11, 2022

View reviewed changes

feat: add schema update support to multiplexing

762f49e

Neenu1995 added the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 11, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 11, 2022

GaoleMeng added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 12, 2022

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 12, 2022

Merge branch 'googleapis:main' into main

de456c2

GaoleMeng added the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 12, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 12, 2022

GaoleMeng merged commit 2adf81b into googleapis:main Nov 12, 2022

release-please bot mentioned this pull request Nov 12, 2022

chore(main): release 2.26.0 #1877

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add update schema support for multiplexing #1867

feat: add update schema support for multiplexing #1867

GaoleMeng commented Nov 5, 2022 •

edited

Loading

yirutang Nov 7, 2022

GaoleMeng Nov 7, 2022

GaoleMeng Nov 10, 2022

yirutang Nov 9, 2022

GaoleMeng Nov 10, 2022

yirutang Nov 9, 2022

GaoleMeng Nov 10, 2022

yirutang Nov 10, 2022

GaoleMeng Nov 10, 2022

yirutang Nov 10, 2022

yirutang Nov 10, 2022

GaoleMeng Nov 11, 2022

yirutang Nov 10, 2022

yirutang Nov 11, 2022

GaoleMeng Nov 11, 2022

feat: add update schema support for multiplexing #1867

feat: add update schema support for multiplexing #1867

Conversation

GaoleMeng commented Nov 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GaoleMeng commented Nov 5, 2022 •

edited

Loading