Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NUTCH-2946 Fetcher: optionally slow down fetching from hosts with repeated exceptions #728

Conversation

sebastian-nagel
Copy link
Contributor

Fetcher to slow down fetching from hosts where requests fail repeatedly with exceptions or HTTP status codes mapped to ProtocolStatus.EXCEPTION (HTTP 403 Forbidden, 429 Too many requests, 5xx server errors, etc.)

… repeatedly

with exceptions or HTTP status codes mapped to ProtocolStatus.EXCEPTION
(HTTP 403 Forbidden, 429 Too many requests, 5xx server errors, etc.)
…eated exceptions

- configure the delay in seconds as a float instead of milliseconds
- use the value of fetcher.server.delay as default
- double the delay with every observed exception (exponential backoff)
  but cap the growth at 2**31 to avoid overflows
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant