Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create run_task python hook to interface with benchmarks #4778

Conversation

waynehamadi
Copy link
Contributor

@waynehamadi waynehamadi commented Jun 22, 2023

Background

Same as this #4772 except we use python method instead of a web server to interface with the benchmarks. And there is no agent or agent insteractions here, just a task.

Changes

  • abstract class task
  • agnostic task, which is the one the benchmark will plug to
  • async challenges so we run the benchmark faster

Documentation

Test Plan

PR Quality Checklist

  • My pull request is atomic and focuses on a single change.
  • I have thoroughly tested my changes with multiple different prompts.
  • I have considered potential risks and mitigations for my changes.
  • I have documented my changes clearly and comprehensively.
  • I have not snuck in any "extra" small tweaks changes.
  • I have run the following commands against my code to ensure it passes our linters:
    black .
    isort .
    mypy
    autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring autogpt tests --in-place

@waynehamadi waynehamadi marked this pull request as draft June 22, 2023 21:00
@netlify
Copy link

netlify bot commented Jun 22, 2023

Deploy Preview for auto-gpt-docs canceled.

Name Link
🔨 Latest commit 685bd30
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs/deploys/64960540029c8a0008a848e7

@Auto-GPT-Bot
Copy link
Contributor

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

@waynehamadi waynehamadi changed the title Create run tasks entry point to interface with benchmarks Create Task.run entry point to interface with benchmarks Jun 22, 2023
@codecov
Copy link

codecov bot commented Jun 22, 2023

Codecov Report

Patch coverage: 73.94% and project coverage change: +0.29 🎉

Comparison is base (32038c9) 70.63% compared to head (685bd30) 70.93%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4778      +/-   ##
==========================================
+ Coverage   70.63%   70.93%   +0.29%     
==========================================
  Files          67       68       +1     
  Lines        3239     3317      +78     
  Branches      514      531      +17     
==========================================
+ Hits         2288     2353      +65     
- Misses        783      787       +4     
- Partials      168      177       +9     
Impacted Files Coverage Δ
autogpt/main.py 23.52% <0.00%> (+1.45%) ⬆️
autogpt/processing/text.py 33.96% <ø> (ø)
autogpt/setup.py 79.71% <ø> (ø)
autogpt/app.py 47.05% <36.36%> (+0.39%) ⬆️
autogpt/json_utils/utilities.py 69.23% <50.00%> (-5.06%) ⬇️
autogpt/llm/utils/__init__.py 49.25% <53.84%> (+0.07%) ⬆️
autogpt/llm/base.py 90.90% <60.00%> (-2.04%) ⬇️
autogpt/prompts/generator.py 86.66% <71.42%> (-3.08%) ⬇️
autogpt/workspace/workspace.py 94.44% <81.25%> (-5.56%) ⬇️
autogpt/llm/providers/openai.py 90.74% <81.48%> (-3.17%) ⬇️
... and 10 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@waynehamadi waynehamadi force-pushed the create-run-task-entry-point branch from 481be6f to ce6bd05 Compare June 22, 2023 21:13
# the workspace directory so we can bind them to the agent.
workspace_directory = Workspace.make_workspace(workspace_directory)
config.workspace_path = str(workspace_directory)
workspace_directory = Workspace.get_workspace_directory(config, workspace_directory)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just created the method so I don't duplicate the method in run task.

Copy link
Member

@Pwuts Pwuts Jul 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This effectively moved the workspace folder from autogpt/auto_gpt_workspace to autogpt/workspace/auto_gpt_workspace, which doesn't correspond with our docker setup and documentation.

@waynehamadi waynehamadi marked this pull request as ready for review June 22, 2023 21:22
@waynehamadi waynehamadi force-pushed the create-run-task-entry-point branch from ce6bd05 to 0327a32 Compare June 22, 2023 21:47
@waynehamadi waynehamadi marked this pull request as draft June 23, 2023 01:02
@waynehamadi waynehamadi force-pushed the create-run-task-entry-point branch from 0327a32 to 5a7dd27 Compare June 23, 2023 19:25
Signed-off-by: Merwane Hamadi <[email protected]>
@waynehamadi waynehamadi force-pushed the create-run-task-entry-point branch from 5a7dd27 to 685bd30 Compare June 23, 2023 20:49
@waynehamadi waynehamadi marked this pull request as ready for review June 23, 2023 21:07
@waynehamadi waynehamadi changed the title Create Task.run entry point to interface with benchmarks Create run_task python hook to interface with benchmarks Jun 24, 2023
@waynehamadi waynehamadi merged commit 222101b into Significant-Gravitas:master Jun 24, 2023
@Pwuts Pwuts mentioned this pull request Jul 21, 2023
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

5 participants