Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hanging indefinitely #2494

Closed
Sourc opened this issue Jun 12, 2023 · 2 comments · Fixed by #2544
Closed

Hanging indefinitely #2494

Sourc opened this issue Jun 12, 2023 · 2 comments · Fixed by #2544
Assignees
Labels
api: spanner Issues related to the googleapis/java-spanner API.

Comments

@Sourc
Copy link

Sourc commented Jun 12, 2023

Hi, this issue is related to the support case 45040288.

A short while ago we had multiple of our services that use Spanner, both with the latest Spanner version at the time (6.42.1) but some using slightly older ones as well, hang indefinitely on this particular row. The only solution when that happened was to restart the service, which would fix the issue - even though it often happened again after a while.

This happened quite often during a two week period towards at least one particular Spanner instance, and we haven't seen it since a couple of weeks now. We have also not been able to determine the exact conditions under which this issue presents itself - We do not believe it to be a recent code change, as we saw it on some services that haven't been updated for months. The code that froze was just a regular resultSet.next() call, preceeded by a singleUse().read(...), with appropriate closing etc.

Since we don't have a reliable way to reproduce this issue, I believe the most straightforward way to solve it would be to implement a timeout mechanic like mentioned in the TODO-comment in the code already.

@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/java-spanner API. label Jun 12, 2023
@rajatbhatta
Copy link
Contributor

Hi @Sourc, thanks for logging the issue!

Do you mean v6.42.1 (and not 6.24.1 in your initial description)? Support case mentions v6.42.1.

@Sourc
Copy link
Author

Sourc commented Jun 13, 2023

Hi, yes a typo. I've corrected it now👍

@rajatbhatta rajatbhatta assigned olavloite and unassigned rajatbhatta Jul 11, 2023
olavloite added a commit that referenced this issue Jul 25, 2023
Use the streamWaitTimeout that has been set on the call context when polling
from the gRPC stream. This prevents the stream from blocking forever if for
some reason the stream is no longer delivering data, and also no error is
propagated to the client.

The default stream wait timeout that is set for all call contexts is 30 mins.
This value can be overridden by configuring a custom call context for a specific
query.

Fixes #2494
olavloite added a commit that referenced this issue Jul 31, 2023
* fix: apply stream wait timeout

Use the streamWaitTimeout that has been set on the call context when polling
from the gRPC stream. This prevents the stream from blocking forever if for
some reason the stream is no longer delivering data, and also no error is
propagated to the client.

The default stream wait timeout that is set for all call contexts is 30 mins.
This value can be overridden by configuring a custom call context for a specific
query.

Fixes #2494

* test: add a wait time to the mock server to ensure that a timeout occurs

* chore: add clirr ignore

* docs: add test + comment for zero timeout

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner Issues related to the googleapis/java-spanner API.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants