Skip to content

Commit

Permalink
fix(bigquery): Make exponential backoff retry second based
Browse files Browse the repository at this point in the history
- Currently these retries are doing retries in milliseconds, which is way too aggressive, the current retry steps are (1ms, 2ms, 4ms, 8ms, 16ms, 32ms, 64ms, 128ms, 256ms, 512ms, 1024ms, ...), it's highly likely the first 7 retries will fail with rate limiting specially in a big workload (specifically this is causing an issue in the Spark bigquery connector), I suspect the exponential retries here were meant to be in seconds (though even if it wasn't meant to, this might be a slightly better approach to reduce the load on the servers
- The new steps would be: 1s, 2s, 4s, 8s, 16s, 32s, 60s (repeated till we exhaust the 5 minutes max retries)
  • Loading branch information
mina-asham committed Jul 27, 2023
1 parent 7d935f0 commit 25ae79a
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,7 @@ private void maybeWaitForInflightQuota() {

@VisibleForTesting
static long calculateSleepTimeMilli(long retryCount) {
return Math.min((long) Math.pow(2, retryCount), 60000);
return Math.min((long) Math.pow(2, retryCount) * 1000, 60000);
}

@VisibleForTesting
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -502,8 +502,8 @@ public void testStreamNameMismatch() throws Exception {

@Test
public void testExponentialBackoff() throws Exception {
assertThat(ConnectionWorker.calculateSleepTimeMilli(0)).isEqualTo(1);
assertThat(ConnectionWorker.calculateSleepTimeMilli(5)).isEqualTo(32);
assertThat(ConnectionWorker.calculateSleepTimeMilli(0)).isEqualTo(1000);
assertThat(ConnectionWorker.calculateSleepTimeMilli(5)).isEqualTo(32000);
assertThat(ConnectionWorker.calculateSleepTimeMilli(100)).isEqualTo(60000);
}

Expand Down

0 comments on commit 25ae79a

Please sign in to comment.