Implement Logging of Self-Feedback in logs/Debug Folder

[AndresCdo]

Background

To improve the performance of our application and understand how it functions in different scenarios, we decided to implement self-feedback logging. This feature will enable the application to log its own feedback regarding its performance and decisions, providing valuable insights for future optimization and development.

Changes

Developed a self-feedback logging mechanism that captures application feedback and appends it to the log file within the logs/DEBUG folder.

Documentation

Descriptions were added in new lines on tests.

Test Plan

I did test it, and it seems to work fine.

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 17, 2023 11:27am

[AndresCdo]

#3842

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.06 🎉

Comparison is base (7c71b43) 62.75% compared to head (0f4858d) 62.82%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3868      +/-   ##
==========================================
+ Coverage   62.75%   62.82%   +0.06%     
==========================================
  Files          73       73              
  Lines        3367     3373       +6     
  Branches      487      487              
==========================================
+ Hits         2113     2119       +6     
  Misses       1107     1107              
  Partials      147      147              

Impacted Files	Coverage Δ
autogpt/agent/agent.py	58.49% <100.00%> (+1.07%)	⬆️
autogpt/log_cycle/log_cycle.py	95.45% <100.00%> (+0.21%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[waynehamadi]

@AndresCdo do it like that:

def get_self_feedback(self, thoughts: dict, llm_model: str) -> str:
        """Generates a feedback response based on the provided thoughts dictionary.
        This method takes in a dictionary of thoughts containing keys such as 'reasoning',
        'plan', 'thoughts', and 'criticism'. It combines these elements into a single
        feedback message and uses the create_chat_completion() function to generate a
        response based on the input message.
        Args:
            thoughts (dict): A dictionary containing thought elements like reasoning,
            plan, thoughts, and criticism.
        Returns:
            str: A feedback response generated using the provided thoughts dictionary.
        """
        ai_role = self.config.ai_role

        feedback_prompt = f"Below is a message from me, an AI Agent, assuming the role of {ai_role}. whilst keeping knowledge of my slight limitations as an AI Agent Please evaluate my thought process, reasoning, and plan, and provide a concise paragraph outlining potential improvements. Consider adding or removing ideas that do not align with my role and explaining why, prioritizing thoughts based on their significance, or simply refining my overall thought process."
        reasoning = thoughts.get("reasoning", "")
        plan = thoughts.get("plan", "")
        thought = thoughts.get("thoughts", "")
        feedback_thoughts = thought + reasoning + plan
        
        messages = {"role": "user", "content": feedback_prompt + feedback_thoughts}
        #log messages
        feedback = create_chat_completion(
            [{"role": "user", "content": feedback_prompt + feedback_thoughts}],
            llm_model,
        )
        #log feedback
        return  feedback

in autogpt/agent/agent.py

1st log call it prompt_self_feedback.json

and the second log call it self_feedback.txt

[AndresCdo]

1st log call it prompt_self_feedback.json

and the second log call it self_feedback.txt

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[Boostrix]

This feature will enable the application to log its own feedback regarding its performance and decisions, providing valuable insights for future optimization and development. [...] Some of the work here may serve as a good basis for #3916 and #15

Indeed, improving self feedback (#3934) could be used as "reward" inside a fitness function (just be sure to provide a "maximize score" objective to the LLM, and pass current score to it as part of the current context).
Especially in combination with user input (human feedback):

• Any time the human isn't saying 'y' but something else, that should impact the fitness function.
• And when inside continuous mode (Allow asking human assistance/feedback incontinuous mode #150), any "error/warning" should affect the reward, too.
• Observing the number of preauthorised commands and their performance, should also affect the fitness function (Show how many auto command remaining when using y -N #1035).
• Human intervention by pausing (Pause continuous loops and give feedback #3083), should negatively affect the reward.
• Letting user feedback time out (Prompting for timed user input in continuous mode #3971) should have NO impact on reward (the agent is probably running unattended or doing just fine).
• The agent prompting for feedback (Proposal for a new command to prompt the user/ ask the user questions. #1548 / Have AI ask question of human #3396) but being told to just continue, should reinforce the current path.
• Likewise increasing the number of preauthorized steps, should positively affect the reward: Prep for #2291 Allow feedback and interrupting during continuous run #2481