Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): retry SignBlob call for URL signing #7862

Merged
merged 32 commits into from
Dec 11, 2024

Conversation

thiyaguk09
Copy link
Contributor

Adds a retry mechanism to the SignBlob call's SignedURL() method to improve reliability in case of transient network or server issues. This change ensures that the SignedURL() operation can be retried under certain conditions, increasing the likelihood of successful URL generation.

@thiyaguk09 thiyaguk09 requested review from a team as code owners November 27, 2024 10:50
@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Nov 27, 2024
@danielduhh danielduhh requested a review from frankyn December 2, 2024 17:45
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
Storage/tests/Unit/SigningHelperTest.php Outdated Show resolved Hide resolved
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
@bshaffer bshaffer self-requested a review December 2, 2024 18:35
using existing retryTrait
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
Storage/tests/Unit/SigningHelperTest.php Outdated Show resolved Hide resolved
Storage/tests/Unit/SigningHelperTest.php Outdated Show resolved Hide resolved
throw new ServiceException('Transient error', 503);
};

$res = $this->helper->proxyPrivateMethodCall('retrySignBlob', [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally speaking, calling private methods is a bad way to test. It would be better to mock the Connection class being passed to v2Sign, in a way that can trigger the retry.

I can work with you on how to write tests like that, if you'd like.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great point. Mocking the Connection class to trigger retries is a much more robust and testable approach. I'd be happy to work with you on implementing this.

Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
review corrections and lint fix
Test case error fix
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
$retryDecider = $this->getRestRetryFunction($resourceName, 'execute', $args);

while ($attempts < $maxRetries) {
$attempts++;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused; shouldn't the retryDecider decide max number of attempts?

Hmm the documentation on this trait is confusing that it uses maxAttempts but doesn't have it in the function signature:
https://github.com/googleapis/google-cloud-php/blob/main/Storage/src/Connection/RetryTrait.php#L190

@bshaffer am I misunderstanding this? i'm not up-to-date on PHP

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I couldn't find a maxRetries or maxAttempts option in the documentation either. That's why I added the maxAttempt directly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll approve from my side; just need @bshaffer to weigh in here when he has a chance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm reading this correctly, the current code does not have any check for maxAttempts, so it seems like this would result in an infinite loop. There is no test case for a retryable error meeting its max attempts. I feel like we definitely need a retry max attempts.

Storage/src/SigningHelper.php Show resolved Hide resolved
@thiyaguk09 thiyaguk09 requested a review from frankyn December 9, 2024 12:21
Storage/src/SigningHelper.php Outdated Show resolved Hide resolved
Storage/tests/Unit/SigningHelperTest.php Show resolved Hide resolved
Storage/tests/Unit/SigningHelperTest.php Outdated Show resolved Hide resolved
implement review changes and maintain code standard
@thiyaguk09 thiyaguk09 requested a review from bshaffer December 10, 2024 04:09
Copy link
Contributor

@bshaffer bshaffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that the logic for "maxAttempts" was intentionally removed, but I think that we still need some kind of protection here. Otherwise the user could get in an infinite loop.

Here's an example of a test I've added that results in an infinite loop, when it should be passing:

    public function testRetrySignBlobThrowsExceptionAfterThreeAttempts()
    {
        $this->expectException(ServiceException::class);
        $this->expectExceptionMessage('Transient error (4 attempts)');

        $attempt = 0;
        $signBlobFn = function () use (&$attempt) {
            throw new ServiceException(
                sprintf('Transient error (%s attempts)', ++$attempt),
                503
            );
        };

        $this->helper->proxyPrivateMethodCall('retrySignBlob', [$signBlobFn]);
    }

I've hardcoded this to have a maxAttempts of 3, but it could be any number.

Ensure proper handling of maxRetries to prevent infinite loop errors.
@thiyaguk09 thiyaguk09 requested a review from bshaffer December 11, 2024 07:08
@bshaffer bshaffer merged commit 4f5cbd5 into googleapis:main Dec 11, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants