feat(backend): Introduce executors shared DB connection #8340

majdyz · 2024-10-15T10:48:55Z

Background

Currently, each execution worker initiates their own DB connection, and the number of DB connections linearly scales with the number of workers. This is problematic when the database we are using has a limited number of DB connections.

Changes 🏗️

Introduced the DatabaseAPI process that exposes prisma queries that will be used by executors in the execution managers.
Removed db connections from ExecutionManager.
Added thread_cached_property for caching pyro client on a thread level.

Testing 🔍

Note

Only for the new autogpt platform, currently in autogpt_platform/

Create from scratch and execute an agent with at least 3 blocks
Import an agent from file upload, and confirm it executes correctly
Upload agent to marketplace
Import an agent from marketplace and confirm it executes correctly
Edit an agent from monitor, and confirm it executes correctly

qodo-merge-pro · 2024-10-15T10:49:23Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Potential Race Condition The new implementation uses a thread-local db_client, which could potentially lead to race conditions if not properly managed across threads. Code Duplication The exposed_run_and_wait decorator is creating wrapper functions for each database operation. This could lead to code duplication and maintenance issues. Complex Type Handling The _pydantic_models_from_type_annotation function has become more complex with the addition of handling for various collection types. This might make it harder to maintain and debug.

netlify · 2024-10-15T10:49:32Z

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	`b8c1a3b`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/670e4a51217403000845a9d9

autogpt_platform/backend/backend/data/credit.py

autogpt_platform/backend/backend/data/graph.py

autogpt_platform/backend/backend/data/queue.py

autogpt_platform/backend/backend/executor/manager.py

autogpt_platform/backend/backend/server/db_api.py

autogpt_platform/backend/backend/server/rest_api.py

autogpt_platform/backend/backend/server/ws_api.py

autogpt_platform/backend/backend/util/cache.py

autogpt_platform/backend/backend/util/retry.py

autogpt_platform/backend/backend/util/service.py

aarushik93 · 2024-10-15T11:29:53Z

this should be against dev

autogpt_platform/backend/backend/server/db_api.py

autogpt_platform/backend/backend/util/service.py

autogpt_platform/backend/backend/util/settings.py

autogpt_platform/backend/backend/data/queue.py

autogpt_platform/backend/poetry.lock

autogpt_platform/backend/backend/executor/manager.py

…t-Gravitas/AutoGPT into zamilmajdy/unified-db-service

* feat(platform): List and revoke credentials in user profile (#8207) Display existing credentials (OAuth and API keys) for all current providers: Google, Github, Notion and allow user to remove them. For providers that support it, we also revoke the tokens through the API: of the providers we currently have, Google and GitHub support it; Notion doesn't. - Add credentials list and `Delete` button in `/profile` - Add `revoke_tokens` abstract method to `BaseOAuthHandler` and implement it in each provider - Revoke OAuth tokens for providers on `DELETE` `/{provider}/credentials/{cred_id}`, and return whether tokens could be revoked - Update `autogpt-server-api/baseClient.ts:deleteCredentials` with `CredentialsDeleteResponse` return type Bonus: - Update `autogpt-server-api/baseClient.ts:_request` to properly handle empty server responses * fix(backend): Lower the number of node workers to save DB connections (#8331) Change [graph]×[node] worker limit from 10×5 to 10×3 --------- Co-authored-by: Reinier van der Leer <[email protected]> * fix(ci,platform): Add dev branch trigger to all ci (#8339) * update ci for dev * update classic * remove duplicate dev * fix(frontend): Fix styling inconsistencies in input elements (#8337) - Apply consistent border styling to `Input`, `Select`, and `Textarea` - Remove `rounded-xl` from node input elements - Add `whitespace-nowrap` to `CustomNode` header category tags --------- Co-authored-by: Zamil Majdy <[email protected]> * feat(builder): Use configmap for builder (#8343) use configmap in builder * fix(platform,infra): Checkin non secret values (#8344) checkin non secrets * security(platform): Add sealed secrets (#8342) * add sealed secrets * add encrypted secrets * remove extra space * Tf public media buckets (#8324) * fix(infra): Fix sealed secret names (#8350) * fix sealed secret names * fix names and add annotation * feat(backend): Introduce executors shared DB connection (#8340) * update health checkendpoint --------- Co-authored-by: Krzysztof Czerwinski <[email protected]> Co-authored-by: Reinier van der Leer <[email protected]> Co-authored-by: Zamil Majdy <[email protected]> Co-authored-by: Swifty <[email protected]>

majdyz added 5 commits October 15, 2024 16:25

feat(backend): Introduced unified backend service

ee2ffab

fix

6bfb1a1

Remove db api from scheduler

7ca9cde

Disabled print to console block

f72c9ad

Revert

da7d35e

majdyz requested review from Swiftyos, Pwuts and aarushik93 October 15, 2024 10:48

majdyz requested a review from a team as a code owner October 15, 2024 10:48

github-actions bot added platform/backend AutoGPT Platform - Back end size/xl labels Oct 15, 2024

qodo-merge-pro bot added the Review effort [1-5]: 4 label Oct 15, 2024

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/data/credit.py Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/data/graph.py Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/data/queue.py Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/executor/manager.py Show resolved Hide resolved

use thread_cached_property

b8c1a3b

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/server/db_api.py Outdated Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/server/rest_api.py Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/server/ws_api.py Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/util/cache.py Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/util/retry.py Show resolved Hide resolved

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/util/service.py Show resolved Hide resolved

Pwuts changed the title ~~feat(platform): Introduce executors shared DB connection~~ feat(backend): Introduce executors shared DB connection Oct 15, 2024

Pwuts reviewed Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/server/db_api.py Outdated Show resolved Hide resolved

majdyz changed the base branch from master to dev October 15, 2024 13:47

reformat

9041fd7

Merge branch 'dev' into zamilmajdy/unified-db-service

fb76ddd

majdyz requested a review from Pwuts October 15, 2024 13:50

majdyz commented Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/util/service.py Show resolved Hide resolved

aarushik93 reviewed Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/util/settings.py Show resolved Hide resolved

aarushik93 reviewed Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/util/settings.py Show resolved Hide resolved

majdyz requested a review from aarushik93 October 15, 2024 14:07

Pwuts reviewed Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/data/queue.py Outdated Show resolved Hide resolved

Pwuts reviewed Oct 15, 2024

View reviewed changes

autogpt_platform/backend/poetry.lock Outdated Show resolved Hide resolved

Pwuts reviewed Oct 15, 2024

View reviewed changes

autogpt_platform/backend/backend/executor/manager.py Outdated Show resolved Hide resolved

majdyz added 3 commits October 15, 2024 17:41

Merge branch 'dev' into zamilmajdy/unified-db-service

236cc5b

reformat

17c4a25

Merge branch 'zamilmajdy/unified-db-service' of github.com:Significan…

3a93bc5

…t-Gravitas/AutoGPT into zamilmajdy/unified-db-service

majdyz requested a review from Pwuts October 15, 2024 14:47

majdyz added 2 commits October 15, 2024 21:51

address comments + format

4bf0362

address comments + format

0340fc4

Swiftyos approved these changes Oct 16, 2024

View reviewed changes

majdyz added 2 commits October 16, 2024 12:38

Merge branch 'dev' into zamilmajdy/unified-db-service

68358b1

Merge branch 'dev' into zamilmajdy/unified-db-service

2e7bb80

majdyz merged commit 6f07d24 into dev Oct 16, 2024
7 checks passed

majdyz deleted the zamilmajdy/unified-db-service branch October 16, 2024 14:15

aarushik93 pushed a commit that referenced this pull request Oct 16, 2024

feat(backend): Introduce executors shared DB connection (#8340)

0f46be2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(backend): Introduce executors shared DB connection #8340

feat(backend): Introduce executors shared DB connection #8340

majdyz commented Oct 15, 2024 •

edited

Loading

qodo-merge-pro bot commented Oct 15, 2024

netlify bot commented Oct 15, 2024 •

edited

Loading

aarushik93 commented Oct 15, 2024

feat(backend): Introduce executors shared DB connection #8340

feat(backend): Introduce executors shared DB connection #8340

Conversation

majdyz commented Oct 15, 2024 • edited Loading

Background

Changes 🏗️

Testing 🔍

qodo-merge-pro bot commented Oct 15, 2024

PR Reviewer Guide 🔍

netlify bot commented Oct 15, 2024 • edited Loading

✅ Deploy Preview for auto-gpt-docs canceled.

aarushik93 commented Oct 15, 2024

majdyz commented Oct 15, 2024 •

edited

Loading

netlify bot commented Oct 15, 2024 •

edited

Loading