New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(agent/workspace): Add GCS and S3 FileWorkspace providers #6485

Merged

Pwuts merged 14 commits into master from agent/add-s3-workspace

Dec 7, 2023

Member

Pwuts commented Dec 3, 2023 •

edited

Loading

Background

We want to be able to run our agent statelessly in cloud environments. This is one piece of the puzzle.

Changes 🏗️

Rename FileWorkspace to LocalFileWorkspace to provide a more descriptive name for the class that represents a file workspace that works with local files.
Create a new base class FileWorkspace to serve as the parent class for LocalFileWorkspace. This allows for easier extension and customization of file workspaces in the future.
Update import statements and references to FileWorkspace throughout the codebase to use the new naming conventions.
Add a GCSFileWorkspace class to support using Google Cloud Storage to store workspace contents.
Add an S3FileWorkspace class to support using AWS or other S3 storage to store workspace contents.
Updated .env.template with new config variables + references to S3 and GCS client libraries' configuration guides.
Modify CI to include a MinIO container to allow testing the S3FileWorkspace.
Add logic to allow switching between Local, GCS and S3 workspace backends.

PR Quality Scorecard ✨

Have you used the PR description template? +2 pts
Is your pull request atomic, focusing on a single change? +5 pts
Have you linked the GitHub issue(s) that this PR addresses? +5 pts
Have you documented your changes clearly and comprehensively? +5 pts
Have you changed or added a feature? -4 pts
- Have you added/updated corresponding documentation? +4 pts
- Have you added/updated corresponding integration tests? +5 pts
Have you changed the behavior of AutoGPT? -5 pts
- Have you also run agbenchmark to verify that these changes do not regress performance? +10 pts


          refactor: Rename FileWorkspace to LocalFileWorkspace and create FileW…

4ae6971

…orkspace abstract class

- Rename `FileWorkspace` to `LocalFileWorkspace` to provide a more descriptive name for the class that represents a file workspace that works with local files.
- Create a new base class `FileWorkspace` to serve as the parent class for `LocalFileWorkspace`. This allows for easier extension and customization of file workspaces in the future.
- Update import statements and references to `FileWorkspace` throughout the codebase to use the new naming conventions.

Pwuts added the function: workspace label

github-actions bot added the size/l label

Pwuts self-assigned this

netlify bot commented Dec 3, 2023 •

edited

Loading

✅ Deploy Preview for auto-gpt-docs ready!

Name	Link
🔨 Latest commit	`e58c4ca`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/6571cbe0ab4f490008b0218f
😎 Deploy Preview	https://deploy-preview-6485--auto-gpt-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Auto-GPT-Bot added the behaviour change label

Contributor

Auto-GPT-Bot commented Dec 3, 2023

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.


          Merge branch 'master' into agent/add-s3-workspace

8d96838

github-actions bot added size/xl and removed size/l labels

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from 91d824d to 152518e Compare

December 3, 2023 16:53

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from 152518e to 2e62580 Compare

December 3, 2023 17:52

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch 2 times, most recently from 6cbfce1 to 0fcd3d8 Compare

December 3, 2023 17:53

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

1 similar comment

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from 0fcd3d8 to 7bfa095 Compare

December 3, 2023 17:59

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from 7bfa095 to 8cbea18 Compare

December 3, 2023 18:04

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from 8cbea18 to f285f91 Compare

December 3, 2023 18:12

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from f285f91 to 6c50de6 Compare

December 3, 2023 18:24

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from 6c50de6 to f1af85f Compare

December 3, 2023 18:32

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts force-pushed the agent/add-s3-workspace branch from f1af85f to 6377d3b Compare

December 3, 2023 18:54


          fix: Improve error handling in S3FileWorkspace.initialize()

f00e2e3

- Do not tolerate all `botocore.exceptions.ClientError`s
- Raise the exception anyways if the error is not "NoSuchBucket"

Pwuts force-pushed the agent/add-s3-workspace branch from 23d9810 to f00e2e3 Compare

December 3, 2023 20:32

Contributor

github-actions bot commented Dec 3, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.


          Merge branch 'master' into agent/add-s3-workspace

a0c895c

Contributor

github-actions bot commented Dec 5, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.


          feat: Add S3 workspace backend support and S3Credentials

d3ab2e7

- Added support for S3 workspace backend in the Autogpt configuration
- Added a new sub-config `S3Credentials` to store S3 credentials
- Modified the `.env.template` file to include variables related to S3 credentials
- Added a new `s3_credentials` attribute on the `Config` class to store S3 credentials
- Moved the `unmasked` method from `ModelProviderCredentials` to the parent `ProviderCredentials` class to handle unmasking for S3 credentials

Contributor

github-actions bot commented Dec 5, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts added 2 commits

December 6, 2023 00:32


          Merge branch 'master' into agent/add-s3-workspace

0447b57


          fix(agent/tests): Fix S3FileWorkspace initialization in test_s3_file_…

3f46c09

…workspace.py

- Update the S3FileWorkspace initialization in the test_s3_file_workspace.py file to include the required S3 Credentials.

Contributor

github-actions bot commented Dec 5, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.


          refactor: Remove S3Credentials and add get_workspace function

04b0af5

- Remove `S3Credentials` as boto3 will fetch the config from the environment by itself
- Add `get_workspace` function in `autogpt.file_workspace` module
- Update `.env.template` and tests to reflect the changes

Contributor

github-actions bot commented Dec 6, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.


          feat(agent/workspace): Make agent workspace backend configurable

f1dfc29

- Modified `autogpt.file_workspace.get_workspace` function to either take a workspace `id` or `root_path`.
- Modified `FileWorkspaceMixin` to use the `get_workspace` function to set up the workspace.
- Updated the type hints and imports accordingly.

Contributor

github-actions bot commented Dec 6, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.


          feat(agent/workspace): Add GCSFileWorkspace for Google Cloud Storage

ba7a91e

- Added support for Google Cloud Storage as a storage backend option in the workspace.
- Created the `GCSFileWorkspace` class to interface with a file workspace stored in a Google Cloud Storage bucket.
- Implemented the `GCSFileWorkspaceConfiguration` class to handle the configuration for Google Cloud Storage workspaces.
- Updated the `get_workspace` function to include the option to use Google Cloud Storage as a workspace backend.
- Added unit tests for the new `GCSFileWorkspace` class.

Contributor

github-actions bot commented Dec 6, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts marked this pull request as ready for review

December 6, 2023 17:43

Pwuts requested a review from a team as a code owner

December 6, 2023 17:43

Pwuts changed the title ~~Add S3 FileWorkspace provider~~ Add GCS and S3 FileWorkspace providers

Pwuts changed the title ~~Add GCS and S3 FileWorkspace providers~~ feat(agent/workspace): Add GCS and S3 FileWorkspace providers


          Merge branch 'master' into agent/add-s3-workspace

9f376eb

Contributor

github-actions bot commented Dec 6, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts commented

View reviewed changes

autogpts/autogpt/autogpt/app/agent_protocol_server.py Outdated Show resolved Hide resolved


          fix: Unbreak use of non-local workspaces in AgentProtocolServer

e58c4ca

- Modify the `_get_task_agent_file_workspace` method to handle both local and non-local workspaces correctly

Contributor

github-actions bot commented Dec 7, 2023

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts added this to the Auto-GPT v0.6.0 milestone

Pwuts removed the behaviour change label

Pwuts merged commit 1f40d72 into master

16 checks passed

Pwuts deleted the agent/add-s3-workspace branch

December 7, 2023 13:46

Jianhao9698 pushed a commit to Jianhao9698/AutoGPT that referenced this pull request


          feat(agent/workspace): Add GCS and S3 FileWorkspace providers (Signif…

fdd2b4b

…icant-Gravitas#6485)

* refactor: Rename FileWorkspace to LocalFileWorkspace and create FileWorkspace abstract class
  - Rename `FileWorkspace` to `LocalFileWorkspace` to provide a more descriptive name for the class that represents a file workspace that works with local files.
  - Create a new base class `FileWorkspace` to serve as the parent class for `LocalFileWorkspace`. This allows for easier extension and customization of file workspaces in the future.
  - Update import statements and references to `FileWorkspace` throughout the codebase to use the new naming conventions.

* feat: Add S3FileWorkspace + tests + test setups for CI and Docker
  - Added S3FileWorkspace class to provide an interface for interacting with a file workspace and storing files in an S3 bucket.
  - Updated pyproject.toml to include dependencies for boto3 and boto3-stubs.
  - Implemented unit tests for S3FileWorkspace.
  - Added MinIO service to Docker CI to allow testing S3 features in CI.
  - Added autogpt-test service config to docker-compose.yml for local testing with MinIO.

* ci(docker): tee test output instead of capturing

* fix: Improve error handling in S3FileWorkspace.initialize()
  - Do not tolerate all `botocore.exceptions.ClientError`s
  - Raise the exception anyways if the error is not "NoSuchBucket"

* feat: Add S3 workspace backend support and S3Credentials
  - Added support for S3 workspace backend in the Autogpt configuration
  - Added a new sub-config `S3Credentials` to store S3 credentials
  - Modified the `.env.template` file to include variables related to S3 credentials
  - Added a new `s3_credentials` attribute on the `Config` class to store S3 credentials
  - Moved the `unmasked` method from `ModelProviderCredentials` to the parent `ProviderCredentials` class to handle unmasking for S3 credentials

* fix(agent/tests): Fix S3FileWorkspace initialization in test_s3_file_workspace.py
  - Update the S3FileWorkspace initialization in the test_s3_file_workspace.py file to include the required S3 Credentials.

* refactor: Remove S3Credentials and add get_workspace function
  - Remove `S3Credentials` as boto3 will fetch the config from the environment by itself
  - Add `get_workspace` function in `autogpt.file_workspace` module
  - Update `.env.template` and tests to reflect the changes

* feat(agent/workspace): Make agent workspace backend configurable
  - Modified `autogpt.file_workspace.get_workspace` function to either take a workspace `id` or `root_path`.
  - Modified `FileWorkspaceMixin` to use the `get_workspace` function to set up the workspace.
  - Updated the type hints and imports accordingly.

* feat(agent/workspace): Add GCSFileWorkspace for Google Cloud Storage
  - Added support for Google Cloud Storage as a storage backend option in the workspace.
  - Created the `GCSFileWorkspace` class to interface with a file workspace stored in a Google Cloud Storage bucket.
  - Implemented the `GCSFileWorkspaceConfiguration` class to handle the configuration for Google Cloud Storage workspaces.
  - Updated the `get_workspace` function to include the option to use Google Cloud Storage as a workspace backend.
  - Added unit tests for the new `GCSFileWorkspace` class.

* fix: Unbreak use of non-local workspaces in AgentProtocolServer
  - Modify the `_get_task_agent_file_workspace` method to handle both local and non-local workspaces correctly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

function: workspace size/xl