-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove databaseRole from metadata DB requests in SpannerIO.ReadChange… #25108
Conversation
Hi, thanks for the quick fix. Could you please first set base branch master, after merged, then cherry pick the merged commit on the master branch into release-2.45.0 branch? This guarantees every commit in release branch is seen on master |
69a51b3
to
5036c64
Compare
I added the commit to master, then cherry-picked it back into release-2.45.0. Is that sufficient, or do I need to create a new pull request? |
Sorry if I was not clear. master means apache/beam:master branch. I've done that for this PR EDIT: not working, need rebase because the current base is on a release branch... |
Assigning reviewers. If you would like to opt out of this review, comment R: @apilloud for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Well despite the base branch issue, shouldn't the correct behavior to be, without setting SpannerConfig.setDatabaseRole(null) the pipeline should just work fine? Sounds weird to me that we need to forcefully overwrite this option there. |
I see the same fix #24177 for another use case. The appear to have same cause. Overwrite it to null here just fix this single use case, but it may again appear elsewhere. Preferably we should fix the underlying logic. |
Run Java_GCP_IO_Direct PreCommit |
Agreed with @Abacn . is there a general case fix instead? |
When reading from change streams there are two databases involved, 1) the primary database that contains the change stream, and 2) a metadata database (maintained by Beam) that holds partition progress state so a job can be restarted and pick up where it left off. To create a spanner config to access the metadata database, the following pattern is used:
The metadata config ( |
Lets create an issue, and have it fixed more robustly. We can merge this workaround into the main branch and this release branch for now |
It looks like the GCP precommit is failing with a timeout. I can't merge when it has failed after retrying it |
Run Java_GCP_IO_Direct PreCommit |
5036c64
to
dbc10db
Compare
@johnjcasey FYI: #25108 (comment) this change was not in master; release-2.45.0 and master has diverged |
yep, we got it merged to master as well |
This change removes the database role from the SpannerConfig that is used to access the Spanner Change Stream metadata database. The role shouldn't be passed to the Change Stream metadata database because there are no roles defined on that database. Addresses #25105.