Improve prompting and prompt generation infrastructure

[Pwuts]

Background

Supporting infrastructure for #5077

Changes

• Clean up prompt generation
• Add debug logging to AIConfig autogeneration
• Clarify prompting and add support for multiple thought processes to Agent

Test Plan

• Tested manually ✅
• CI

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes.
• I have run the following commands against my code to ensure it passes our linters:

  black .
  isort .
  mypy
  autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring autogpt tests --in-place

[netlify]

❌ Deploy Preview for auto-gpt-docs failed.

Name	Link
🔨 Latest commit	f06b756
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/64c54308ea115600085d1f48

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[codecov]

Codecov Report

Patch coverage: 90.32% and project coverage change: +0.03% 🎉

Comparison is base (c8d9b69) 51.18% compared to head (f06b756) 51.21%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5076      +/-   ##
==========================================
+ Coverage   51.18%   51.21%   +0.03%     
==========================================
  Files         116      116              
  Lines        4937     4938       +1     
  Branches      663      664       +1     
==========================================
+ Hits         2527     2529       +2     
+ Misses       2216     2214       -2     
- Partials      194      195       +1     

Files Changed	Coverage Δ
autogpt/config/ai_config.py	73.68% <75.00%> (-2.59%)	⬇️
autogpt/agents/base.py	71.75% <86.36%> (+2.52%)	⬆️
autogpt/agents/agent.py	50.00% <100.00%> (ø)
autogpt/app/setup.py	73.41% <100.00%> (+0.34%)	⬆️
autogpt/config/prompt_config.py	76.47% <100.00%> (ø)
autogpt/llm/utils/__init__.py	46.03% <100.00%> (+0.87%)	⬆️
autogpt/prompts/generator.py	95.45% <100.00%> (+2.47%)	⬆️
autogpt/prompts/prompt.py	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.