Add command for directly executing python code

[erik-megarad]

Background

When running Auto-GPT it very, very frequently wants to run python code. To do it is usually 3 steps (execute file [that probably does not exist], write file, execute file). Not only is this quite costly, but if there are errors in the code (there usually are), the iteration cycle to fix it is slow and extremely error prone.

This change adds a new command, execute_python_code, which will execute straight python code. It does this by writing the code to a file in the workspace directory, executing it using the existing execute_python_file command, and then returning the results. The file is kept around for later reference.

I know that we're limiting the number of new commands we have, but this one is a 200%+ increase in efficiency for the majority of tasks that Auto-GPT decides to do.

Changes

• Add a new command, execute_python_code

Documentation

• In-code comments

Test Plan

• Wrote a new unit test
• Ran a few manual tests, output of one example below:

Input:what is the square root of 213949
SYSTEM:  Human feedback: what is the square root of 213949
MATHGPT THOUGHTS:  I will use Python to calculate the square root of 213949.
REASONING:  Python has a built-in math library that can be used to calculate the square root of a number.
PLAN:
-  Use Python to calculate the square root of 213949
-  Provide the result to the user
CRITICISM:  I need to ensure that the result is accurate and that I am using the correct syntax to calculate the square root.
SPEAK:  I will use Python to calculate the square root of 213949.
NEXT ACTION:  COMMAND = execute_python_code ARGUMENTS = {'code': 'import math\nresult = math.sqrt(213949)\nprint(result)'}
  Enter 'y' to authorise command, 'y -N' to run N continuous commands, 's' to run self-feedback commands, 'n' to exit program, or enter feedback for MathGPT...
  Asking user via keyboard...
Input:y
-=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-=
  Executing python file '/app/autogpt/auto_gpt_workspace/temp_code.py' in working directory '/app/autogpt/auto_gpt_workspace'
SYSTEM:  Command execute_python_code returned: 462.5462139073241

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes.
• I have run the following commands against my code to ensure it passes our linters:

  black .
  isort .
  mypy
  autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports autogpt tests --in-place

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 25 minutes (more than 100, code: "api-deployments-free-per-day").

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 20 minutes (more than 100, code: "api-deployments-free-per-day").

[erik-megarad]

Since this is a new command that will be highly used I've opened a discussion in discord. Will update this PR with any outcome. Also welcome comments directly on the PR

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[codecov]

Codecov Report

Patch coverage: 84.61% and project coverage change: +0.11 🎉

Comparison is base (3b0d49a) 69.67% compared to head (855edac) 69.79%.

❗ Current head 855edac differs from pull request most recent head 754b993. Consider uploading reports for the commit 754b993 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4581      +/-   ##
==========================================
+ Coverage   69.67%   69.79%   +0.11%     
==========================================
  Files          72       72              
  Lines        3558     3575      +17     
  Branches      569      570       +1     
==========================================
+ Hits         2479     2495      +16     
- Misses        890      891       +1     
  Partials      189      189              

Impacted Files	Coverage Δ
autogpt/commands/execute_code.py	68.93% <84.61%> (+3.37%)	⬆️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)			Jun 9, 2023 5:34pm

[waynehamadi]

@erik-megarad we skipped the debug code challenge, we need to unskip it so we can tie the challenges to this command.
This is not a blocker to merge the PR but it would be nice. I will do it when I am done with my current tasks.

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[Boostrix]

Requesting review by @Pwuts since he's commented on this idea on discord and in related RFEs, and indicated being supportive of this.

I really believe we should strive to get this reviewed and integrated, we have a number of requests related to creating commands (#4549) and tooling (#4214) "on the fly", being able to execute python code directly (including potentially unit tests), could be a first step towards this.

This would also help implement other features, and make a few similar PRs obsolete:

• Calculations and math #3412
• Calculate [expression] command / plugin #3775

We would only need a way to register certain code as [new] commands.

Related:

• Executing python code directly from AI output #286
• [DRAFT] allow python code only to be compiled/validated #4112
• Auto-discover new capabilities via API's and dynamic run-times. #56

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 3 hours (more than 100, code: "api-deployments-free-per-day").

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 2 hours (more than 100, code: "api-deployments-free-per-day").

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 38 minutes (more than 100, code: "api-deployments-free-per-day").

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 13 minutes (more than 100, code: "api-deployments-free-per-day").

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 6 hours (more than 100, code: "api-deployments-free-per-day").

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[Pwuts]

Why did you go with basename rather than filename?

[erik-megarad]

I explained it in a previous comment. Using filename caused the AI to always give an absolute path, no matter what I did. Since we’re placing it specifically in another location it was necessary to change the argument name.

…

On Fri, Jun 9, 2023 at 10:48 AM Reinier van der Leer < ***@***.***> wrote: Why did you do with basename rather than filename? — Reply to this email directly, view it on GitHub <#4581 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABQXHCPMNPYMWGBBZQJN3XKNOVXANCNFSM6AAAAAAY2I2NEM> . You are receiving this because you were mentioned.Message ID: ***@***.***>