Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(bigquery): Make exponential backoff retry second based
- Currently these retries are doing retries in milliseconds, which is way too aggressive, the current retry steps are (1ms, 2ms, 4ms, 8ms, 16ms, 32ms, 64ms, 128ms, 256ms, 512ms, 1024ms, ...), it's highly likely the first 7 retries will fail with rate limiting specially in a big workload (specifically this is causing an issue in the Spark bigquery connector), I suspect the exponential retries here were meant to be in seconds (though even if it wasn't meant to, this might be a slightly better approach to reduce the load on the servers - The new steps would be: 1s, 2s, 4s, 8s, 16s, 32s, 60s (repeated till we exhaust the 5 minutes max retries)
- Loading branch information