Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #4461: Don't record error requests in challenges #4469

Conversation

erik-megarad
Copy link
Contributor

Background

As described in #4461, we don't want VCR to record error responses (e.g. quota limit errors) when recording challenges. This PR will filter error requests for challenges only.

Changes

  • Extract the base VCR config into a variable so we don't have to copy+paste the config (resilient to changes to base config)
  • Change the VCR config for challenges to filter errors

Documentation

  • One in-line comment

Test Plan

  • Set my usage limit in OpenAI to 1
  • Ran a challenge
  • Ensured that the error request was not included

PR Quality Checklist

  • My pull request is atomic and focuses on a single change.
  • I have thoroughly tested my changes with multiple different prompts.
  • I have considered potential risks and mitigations for my changes.
  • I have documented my changes clearly and comprehensively.
  • I have not snuck in any "extra" small tweaks changes.
  • I have run the following commands against my code to ensure it passes our linters:
    black .
    isort .
    mypy
    autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports autogpt tests --in-place

@vercel
Copy link

vercel bot commented May 29, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
docs ⬜️ Ignored (Inspect) May 30, 2023 2:26pm

@erik-megarad erik-megarad force-pushed the improvement/challenge_error_vcr branch from c7481a7 to e7fea50 Compare May 29, 2023 21:14
@codecov
Copy link

codecov bot commented May 29, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (56650b6) 68.57% compared to head (c7481a7) 68.57%.

❗ Current head c7481a7 differs from pull request most recent head 496ed2b. Consider uploading reports for the commit 496ed2b to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4469   +/-   ##
=======================================
  Coverage   68.57%   68.57%           
=======================================
  Files          72       72           
  Lines        3520     3520           
  Branches      561      561           
=======================================
  Hits         2414     2414           
  Misses        921      921           
  Partials      185      185           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@waynehamadi
Copy link
Contributor

@erik-megarad I will add that change, there is some adjustment to make thank you very much !

@waynehamadi waynehamadi force-pushed the improvement/challenge_error_vcr branch from c0c80d1 to 1d8f934 Compare May 30, 2023 13:12
@waynehamadi
Copy link
Contributor

Set my usage limit in OpenAI to 1 => love this

@waynehamadi
Copy link
Contributor

@erik-megarad you're on discord ? If not join us:
https://discord.gg/autogpt

@waynehamadi waynehamadi force-pushed the improvement/challenge_error_vcr branch from 1d8f934 to e7fea50 Compare May 30, 2023 14:26
@waynehamadi waynehamadi merged commit b56352e into Significant-Gravitas:master May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants