feat(agent): Abstract file storage access #6931

kcze · 2024-02-28T11:16:02Z

This improvement abstracts file access across whole project, so that AutoGPT app/agents can use different backends (e.g. local, GCS, S3).

Fixes related issue: OPEN-103

Changes

⚠️ BREAKING: Rename WORKSPACE_BACKEND to FILE_STORAGE_BACKEND
⚠️ BREAKING: Rename WORKSPACE_STORAGE_BUCKET to STORAGE_BUCKET
Rename FileWorkspace to FileStorage
- autogpt.file_workspace -> autogpt.file_storage
- LocalFileWorkspace -> LocalFileStorage
- S3FileWorkspace -> S3FileStorage
- GCSFileWorkspace -> GCSFileStorage
Add FileStorage method clone_with_subroot
Add FileStorage methods exists, make_dir, delete_dir, rename, list_files, list_folders
Create AgentFileManagerMixin to replace AgentFileManager, FileWorkspaceMixin, BaseAgent.attach_fs(..)
- Replace BaseAgentSettings.save_to_json_file(..) method by AgentFileManagerMixin.save_state()
- Replace BaseAgent.set_id(..) method by AgentFileManagerMixin.change_agent_id(..)
- Remove BaseAgentSettings.load_from_json_file(..)
- Remove AgentSettings.agent_data_dir
Updates across codebase to adjust for this change, such as replacing direct file access to use FileStorage instead
- Rewrite AgentManager to use FileStorage rather than direct local file access
  - Rename AgentManager.retrieve_state(..) method to load_agent_state
  - Add docstrings to AgentManager
- Update AgentProtocolServer to work with the new FileStorage system and AgentFileManagerMixin
- Update autogpt.commands.file_operations to use FileStorage and AgentFileManagerMixin features
Make agent_id and file_storage parameters for creating an Agent:
- Update autogpt.agent_factory.configurators:
  create_agent(..), configure_agent_with_state(..), _configure_agent(..), create_agent_state(..)
- Update autogpt.agent_factory.generators:
  generate_agent_for_task(..)
- Update Agent.__init__(..)
- Update BaseAgent.__init__(..)
- Initialize and pass in file_storage in autogpt.app.main.run_auto_gpt(..) and autogpt.app.main.run_auto_gpt_server(..)
Update tests for FileStorage implementations and usages
Rename workspace fixture to storage
- Update conftest.py

Add list_folders(..) and update exists(..) to include for folders.

Merge AgentFileManager and FileWorkspaceMixin

Change multiple files to use FileManager and FileStorage

github-actions · 2024-02-28T11:16:21Z

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

netlify · 2024-02-28T11:17:03Z

✅ Deploy Preview for auto-gpt-docs ready!

Name	Link
🔨 Latest commit	`480002f`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/65ef7688ac23930008d86074
😎 Deploy Preview	https://deploy-preview-6931--auto-gpt-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

github-actions · 2024-02-28T11:29:48Z

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

github-actions · 2024-02-28T11:32:01Z

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

Pwuts

For a first XL pull request this looks very good! I do have a number of comments/requests but don't let that be a discouragement. :)

Questions:

Have the added methods on GCSFileStorage and S3FileStorage been tested?
How much of this has been end-to-end tested?

autogpts/autogpt/agbenchmark_config/benchmarks.py

autogpts/autogpt/autogpt/agent_manager/agent_manager.py

autogpts/autogpt/autogpt/agents/features/agent_file_manager.py

autogpts/autogpt/autogpt/file_storage/base.py

autogpts/autogpt/autogpt/file_storage/gcs.py

autogpts/autogpt/tests/utils.py

autogpts/autogpt/autogpt/app/main.py

autogpts/autogpt/autogpt/agents/utils/file_manager.py

autogpts/autogpt/autogpt/agents/features/agent_file_manager.py

autogpts/autogpt/autogpt/app/main.py

Code cleanup, import order, better naming.

github-actions · 2024-03-01T13:09:14Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2024-03-01T16:24:47Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2024-03-06T13:39:06Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

autogpts/autogpt/tests/conftest.py

autogpts/autogpt/tests/unit/test_gcs_file_storage.py

autogpts/autogpt/autogpt/file_storage/gcs.py

autogpts/autogpt/autogpt/agents/agent.py

autogpts/autogpt/autogpt/agents/base.py

autogpts/autogpt/autogpt/app/agent_protocol_server.py

github-actions · 2024-03-07T22:03:51Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2024-03-07T22:06:29Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2024-03-11T11:46:00Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Pwuts

Changes look good and tests pass. LGTM!

autogpts/autogpt/autogpt/agents/features/agent_file_manager.py

autogpts/autogpt/test_file.txt

github-actions · 2024-03-11T21:24:42Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Pwuts · 2024-03-12T14:56:37Z

Fails to deploy to GCP:

ERROR 2024-03-12T14:47:31.371921Z Traceback (most recent call last): File "<string>", line 1, in <module> File "/venv/agpt-9TtSrW0h-py3.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
DEFAULT 2024-03-12T14:47:31.371932Z return self.main(*args, **kwargs)
DEFAULT 2024-03-12T14:47:31.371941Z File "/venv/agpt-9TtSrW0h-py3.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
DEFAULT 2024-03-12T14:47:31.371945Z rv = self.invoke(ctx)
DEFAULT 2024-03-12T14:47:31.371951Z File "/venv/agpt-9TtSrW0h-py3.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
DEFAULT 2024-03-12T14:47:31.371955Z return _process_result(sub_ctx.command.invoke(sub_ctx))
DEFAULT 2024-03-12T14:47:31.371959Z File "/venv/agpt-9TtSrW0h-py3.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
DEFAULT 2024-03-12T14:47:31.371963Z return ctx.invoke(self.callback, **ctx.params)
DEFAULT 2024-03-12T14:47:31.371968Z File "/venv/agpt-9TtSrW0h-py3.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
DEFAULT 2024-03-12T14:47:31.371972Z return __callback(*args, **kwargs)
DEFAULT 2024-03-12T14:47:31.371976Z File "/app/autogpt/app/cli.py", line 272, in serve
DEFAULT 2024-03-12T14:47:31.371980Z run_auto_gpt_server(
DEFAULT 2024-03-12T14:47:31.371984Z File "/app/autogpt/core/runner/client_lib/utils.py", line 60, in wrapper
DEFAULT 2024-03-12T14:47:31.371988Z return asyncio.run(f(*args, **kwargs))
DEFAULT 2024-03-12T14:47:31.371992Z File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
DEFAULT 2024-03-12T14:47:31.371995Z return loop.run_until_complete(main)
DEFAULT 2024-03-12T14:47:31.372Z File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
DEFAULT 2024-03-12T14:47:31.372003Z return future.result()
DEFAULT 2024-03-12T14:47:31.372007Z File "/app/autogpt/app/main.py", line 351, in run_auto_gpt_server
DEFAULT 2024-03-12T14:47:31.372011Z file_storage = get_storage(
DEFAULT 2024-03-12T14:47:31.372015Z File "/app/autogpt/file_storage/__init__.py", line 37, in get_storage
DEFAULT 2024-03-12T14:47:31.372019Z return GCSFileStorage(config)
DEFAULT 2024-03-12T14:47:31.372023Z File "/app/autogpt/file_storage/gcs.py", line 36, in __init__
DEFAULT 2024-03-12T14:47:31.372026Z assert self._root.is_absolute()
DEFAULT 2024-03-12T14:47:31.372043Z AssertionError

kcze added 13 commits February 21, 2024 23:52

Add exists and make_dir to FileWorkspace

6b34f50

WIP Update file_manager backend

edf410b

Rename *FileWorkspace* to *FileStorage*

7ae2c3b

Update FileStorage

902207b

Add list_folders(..) and update exists(..) to include for folders.

Add FileManager

ab80a74

Add AgentFileManagerMixin

e16115e

Merge AgentFileManager and FileWorkspaceMixin

Update codebase

898083e

Change multiple files to use FileManager and FileStorage

Fix file_operations

78ab5bd

Fix tests

65d5122

Fix agent_id override

3395903

Cache logs in AgentFileManagerMixin

d260568

Add comments

d781dc9

Merge branch 'master' into kpczerwinski/open-194...

23e01b6

kcze requested a review from a team as a code owner February 28, 2024 11:16

github-actions bot added the size/xl label Feb 28, 2024

Fix linting

ff3e39d

Remove unused import

e4a6dea

Pwuts requested changes Feb 28, 2024

View reviewed changes

Pwuts added the function: workspace label Feb 28, 2024

Pwuts changed the title ~~OPEN-194: Improvement - abstract storage access~~ feat(agent): Abstract storage access Feb 28, 2024

Minor changes

896ae16

Code cleanup, import order, better naming.

github-actions bot added the Classic AutoGPT Agent label Mar 1, 2024

Add clone_with_subroot to FileStorage

29a7dfb

Significant-Gravitas deleted a comment from github-actions bot Mar 1, 2024

kcze added 2 commits March 1, 2024 12:36

Remove FileManager

8963e68

Add rename to FileStorage

ca93682

kcze added 2 commits March 1, 2024 14:19

Remove unused agent_data_dir

9d672cd

Update tests

49cda97

kcze added 2 commits March 5, 2024 23:07

Update S3FileStorage and tests

1ce1eff

Update GCSFileStorage and tests

2e73084

kcze requested a review from Pwuts March 6, 2024 15:42

Pwuts requested changes Mar 7, 2024

View reviewed changes

Pwuts changed the title ~~feat(agent): Abstract storage access~~ feat(agent): Abstract file storage access Mar 7, 2024

kcze added 3 commits March 7, 2024 21:37

Fix integration tests

04d3bb0

Fix rename in GCSFileStorage

1c714f1

Fix AgentProtocolServer

2548732

Remove unused import

bb0b65e

kcze requested a review from Pwuts March 7, 2024 22:08

Revert unrelated formatting

ea2becf

Pwuts previously approved these changes Mar 11, 2024

View reviewed changes

Pwuts reviewed Mar 11, 2024

View reviewed changes

autogpts/autogpt/autogpt/agents/features/agent_file_manager.py Show resolved Hide resolved

Pwuts reviewed Mar 11, 2024

View reviewed changes

autogpts/autogpt/test_file.txt Outdated Show resolved Hide resolved

yeet test_file.txt

480002f

Pwuts dismissed their stale review via 480002f March 11, 2024 21:24

Pwuts merged commit 37904a0 into Significant-Gravitas:master Mar 11, 2024
11 of 14 checks passed

kcze deleted the kpczerwinski/open-194-abstract-storage-access branch March 11, 2024 21:42

This was referenced Mar 18, 2024

fix(agent): Make save_state behave like save as #7025

Merged

feat(agent): Upgrade commands system #7030

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(agent): Abstract file storage access #6931

feat(agent): Abstract file storage access #6931

kcze commented Feb 28, 2024 •

edited by Pwuts

Loading

github-actions bot commented Feb 28, 2024

netlify bot commented Feb 28, 2024 •

edited

Loading

github-actions bot commented Feb 28, 2024

github-actions bot commented Feb 28, 2024

Pwuts left a comment

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 6, 2024

github-actions bot commented Mar 7, 2024

github-actions bot commented Mar 7, 2024

github-actions bot commented Mar 11, 2024

Pwuts left a comment

github-actions bot commented Mar 11, 2024

Pwuts commented Mar 12, 2024

feat(agent): Abstract file storage access #6931

feat(agent): Abstract file storage access #6931

Conversation

kcze commented Feb 28, 2024 • edited by Pwuts Loading

Changes

github-actions bot commented Feb 28, 2024

netlify bot commented Feb 28, 2024 • edited Loading

✅ Deploy Preview for auto-gpt-docs ready!

github-actions bot commented Feb 28, 2024

github-actions bot commented Feb 28, 2024

Pwuts left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 6, 2024

github-actions bot commented Mar 7, 2024

github-actions bot commented Mar 7, 2024

github-actions bot commented Mar 11, 2024

Pwuts left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 11, 2024

Pwuts commented Mar 12, 2024

kcze commented Feb 28, 2024 •

edited by Pwuts

Loading

netlify bot commented Feb 28, 2024 •

edited

Loading