-
Notifications
You must be signed in to change notification settings - Fork 44.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(backend): Fix .env file read contention on pyro connection setup #8736
Conversation
…tion-to-async' into zamilmajdy/revert-converted-function-to-async
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
✅ Deploy Preview for auto-gpt-docs canceled.
|
@majdyz is this related to today's outage? Because we don't read from .env |
@aarushik93 unrelated, just something I found during the local test. |
Can you explain a bit more, the PR description doesn't really explain to me what the issue is or what its fixing |
@aarushik93 Described in the PR like this:
Let me know if there is any part that needs to be clarified. |
Gotcha, thanks that clears it up for me! |
On high concurrent pyro connections initialization, the .env file being on the high read contention causing connection establishment to fail.
Calling
Secrets()
,Config()
, andSettings()
, will always trigger a file read. And when this is executed too many times it will cause read contention and pass the file descriptor limit set by the OS. This will trigger retry and causes delay & thundering herd issues. In Unix this is errored out as:Too Many Open Files
orThere are only ... file descriptors (hard limit) available
To reproduce the error you can try to do the run locally and run hundreds of concurrent requests.
Changes 🏗️
Move all the usage of Config as a globally initialized variable so it's only called once per process.
Checklist 📋
For code changes:
Example test plan
For configuration changes:
.env.example
is updated or already compatible with my changesdocker-compose.yml
is updated or already compatible with my changesExamples of configuration changes