Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(backend): Lower the number of graph and node workers #8331

Merged
merged 3 commits into from
Oct 14, 2024
Merged

fix(backend): Lower the number of graph and node workers #8331

merged 3 commits into from
Oct 14, 2024

Conversation

aarushik93
Copy link
Contributor

Background

Supabase limits the connections available to pg. Execution manager doesnt release nor pool connections so we keep running out.

Solution https://discord.com/channels/1126875755960336515/1285234928367308821/1294302355885391916 from Zamil

Changes 🏗️

Change the graph & node workers to 4,2 respectively

Testing 🔍

Note

Only for the new autogpt platform, currently in autogpt_platform/

  • Create from scratch and execute an agent with at least 3 blocks
  • Import an agent from file upload, and confirm it executes correctly
  • Upload agent to marketplace
  • Import an agent from marketplace and confirm it executes correctly
  • Edit an agent from monitor, and confirm it executes correctly

@aarushik93 aarushik93 requested a review from a team as a code owner October 14, 2024 12:34
@github-actions github-actions bot added the platform/backend AutoGPT Platform - Back end label Oct 14, 2024
@qodo-merge-pro Qodo Merge Pro
Copy link

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Performance Impact
Reducing the number of graph and node workers may impact system performance and throughput. Verify that this change doesn't negatively affect the system's ability to handle the expected workload.

Configuration Consistency
Ensure that the reduction in worker numbers is appropriate for the production environment and doesn't lead to performance bottlenecks under high load.

@netlify Netlify
Copy link

netlify bot commented Oct 14, 2024

✅ Deploy Preview for auto-gpt-docs canceled.

Name Link
🔨 Latest commit 6b2ba4b
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs/deploys/670d0fc9856c640008ac3448

Swiftyos
Swiftyos previously approved these changes Oct 14, 2024
View reviewed changes
@aarushik93 aarushik93 changed the base branch from master to dev October 14, 2024 14:39
Pwuts
Pwuts approved these changes Oct 14, 2024
View reviewed changes
Copy link
Member

@Pwuts Pwuts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing from 10×5 to 10×3 will hopefully leave enough available connections for the rest of the system without compromising UX.

@Pwuts Pwuts enabled auto-merge (squash) October 14, 2024 15:58
@Pwuts Pwuts changed the title fix(platform): Lower the number of graph and node workers fix(backend): Lower the number of graph and node workers Oct 14, 2024
@Pwuts Pwuts merged commit 1c71351 into dev Oct 14, 2024
6 checks passed
@Pwuts Pwuts deleted the aarushikansal/update-workers branch October 14, 2024 16:00
aarushik93 added a commit that referenced this pull request Oct 16, 2024
@aarushik93 @kcze
@Pwuts @majdyz @Swiftyos
fix(infra): Update backend server health check endpoint (#8351)
405dd16 
* feat(platform): List and revoke credentials in user profile (#8207)

Display existing credentials (OAuth and API keys) for all current providers: Google, Github, Notion and allow user to remove them. For providers that support it, we also revoke the tokens through the API: of the providers we currently have, Google and GitHub support it; Notion doesn't.

- Add credentials list and `Delete` button in `/profile`
- Add `revoke_tokens` abstract method to `BaseOAuthHandler` and implement it in each provider
- Revoke OAuth tokens for providers on `DELETE` `/{provider}/credentials/{cred_id}`, and return whether tokens could be revoked
   - Update `autogpt-server-api/baseClient.ts:deleteCredentials` with `CredentialsDeleteResponse` return type

Bonus:
- Update `autogpt-server-api/baseClient.ts:_request` to properly handle empty server responses

* fix(backend): Lower the number of node workers to save DB connections (#8331)

Change [graph]×[node] worker limit from 10×5 to 10×3

---------

Co-authored-by: Reinier van der Leer <[email protected]>

* fix(ci,platform): Add dev branch trigger to all ci (#8339)

* update ci for dev

* update classic

* remove duplicate dev

* fix(frontend): Fix styling inconsistencies in input elements (#8337)

- Apply consistent border styling to `Input`, `Select`, and `Textarea`
   - Remove `rounded-xl` from node input elements

- Add `whitespace-nowrap` to `CustomNode` header category tags

---------

Co-authored-by: Zamil Majdy <[email protected]>

* feat(builder): Use configmap for builder (#8343)

use configmap in builder

* fix(platform,infra): Checkin non secret values  (#8344)

checkin non secrets

* security(platform): Add sealed secrets (#8342)

* add sealed secrets

* add encrypted secrets

* remove extra space

* Tf public media buckets (#8324)

* fix(infra): Fix sealed secret names  (#8350)

* fix sealed secret names

* fix names and add annotation

* feat(backend): Introduce executors shared DB connection (#8340)

* update health checkendpoint

---------

Co-authored-by: Krzysztof Czerwinski <[email protected]>
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: Zamil Majdy <[email protected]>
Co-authored-by: Swifty <[email protected]>
aarushik93 added a commit that referenced this pull request Oct 16, 2024
@aarushik93 @kcze
@Pwuts @majdyz @Swiftyos
fix(infra): Update backend server health check endpoint (#8351)
f621d39 
* feat(platform): List and revoke credentials in user profile (#8207)

Display existing credentials (OAuth and API keys) for all current providers: Google, Github, Notion and allow user to remove them. For providers that support it, we also revoke the tokens through the API: of the providers we currently have, Google and GitHub support it; Notion doesn't.

- Add credentials list and `Delete` button in `/profile`
- Add `revoke_tokens` abstract method to `BaseOAuthHandler` and implement it in each provider
- Revoke OAuth tokens for providers on `DELETE` `/{provider}/credentials/{cred_id}`, and return whether tokens could be revoked
   - Update `autogpt-server-api/baseClient.ts:deleteCredentials` with `CredentialsDeleteResponse` return type

Bonus:
- Update `autogpt-server-api/baseClient.ts:_request` to properly handle empty server responses

* fix(backend): Lower the number of node workers to save DB connections (#8331)

Change [graph]×[node] worker limit from 10×5 to 10×3

---------

Co-authored-by: Reinier van der Leer <[email protected]>

* fix(ci,platform): Add dev branch trigger to all ci (#8339)

* update ci for dev

* update classic

* remove duplicate dev

* fix(frontend): Fix styling inconsistencies in input elements (#8337)

- Apply consistent border styling to `Input`, `Select`, and `Textarea`
   - Remove `rounded-xl` from node input elements

- Add `whitespace-nowrap` to `CustomNode` header category tags

---------

Co-authored-by: Zamil Majdy <[email protected]>

* feat(builder): Use configmap for builder (#8343)

use configmap in builder

* fix(platform,infra): Checkin non secret values  (#8344)

checkin non secrets

* security(platform): Add sealed secrets (#8342)

* add sealed secrets

* add encrypted secrets

* remove extra space

* Tf public media buckets (#8324)

* fix(infra): Fix sealed secret names  (#8350)

* fix sealed secret names

* fix names and add annotation

* feat(backend): Introduce executors shared DB connection (#8340)

* update health checkendpoint

---------

Co-authored-by: Krzysztof Czerwinski <[email protected]>
Co-authored-by: Reinier van der Leer <[email protected]>
Co-authored-by: Zamil Majdy <[email protected]>
Co-authored-by: Swifty <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Pwuts Pwuts Pwuts approved these changes

@Swiftyos Swiftyos Swiftyos left review comments

Assignees

@Pwuts Pwuts

Labels
platform/backend AutoGPT Platform - Back end Review effort [1-5]: 1 size/s
Projects
AutoGPT development kanban
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@aarushik93 @Swiftyos @Pwuts