Skip to content

Issues: UKGovernmentBEIS/inspect_ai

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

28 Open 215 Closed
28 Open 215 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

docker provider write_file() hanging in docker compose cp
#1034 opened Dec 21, 2024 by jjallaire
1
Scoring crashes in transcript JSON change tracking on tasks involving model grading bug Something isn't working
#1015 opened Dec 17, 2024 by MSchmatzAISI
4
GAIA web browser tool returns <empty> when affected by packet loss
#988 opened Dec 12, 2024 by alex-treebeard
viewer: display cost estimates in usage tab
#980 opened Dec 10, 2024 by jjallaire-aisi
OpenAI model provider doesn't handle invalid_request_error
#968 opened Dec 9, 2024 by skinnerjc
eval_set only retries "cancelled" or "errored" logs
#913 opened Nov 28, 2024 by max-kaufmann
7
support fractional unicode code points (e.g. ½, ¾) in numeric normalisation.
#897 opened Nov 25, 2024 by jjallaire
Add more documentation on CLI
#868 opened Nov 20, 2024 by mrahtz
1
[Feature Request] Add user_message solver
#864 opened Nov 19, 2024 by kaifronsdal
1
Log a "Sample error detected" if a sample has an error, so that we can see it in the console when we are running an eval
#824 opened Nov 8, 2024 by max-kaufmann
For certain Model API calls, if they error we don't have the API call logged to the transcript.
#823 opened Nov 8, 2024 by max-kaufmann
1
[Question] Custom metric type
#775 opened Oct 30, 2024 by us2547
2
@dragonstyle
6
Adding a scorer without rerunning existing scorers.
#770 opened Oct 27, 2024 by sohaibimran7
2
Failed epoch doesn't stop rest of task's epochs
#746 opened Oct 23, 2024 by craigwalton-dsit
Web browser struggles with large pages web browser
#740 opened Oct 21, 2024 by art-dsit
1
plot_results(): Are there any frameworks that allow summarising and visualising inspect logs?
#704 opened Oct 16, 2024 by sohaibimran7
2
support for web browser navigations that open new windows web browser
#681 opened Oct 10, 2024 by jjallaire-aisi
@MariaIzobava
Error running inspect score with a scoring function that uses a task-level sandbox
#679 opened Oct 10, 2024 by bienehito
[Feature request] command line flag to capture the values of locals in the printed traceback
#673 opened Oct 9, 2024 by JasonGross
1
.json file is missing all messages when an error is thrown
#672 opened Oct 9, 2024 by JasonGross
1
Web Browser tool cannot see checkbox state web browser
#636 opened Oct 3, 2024 by sdtblckgov
@MariaIzobava
Feature Request: Support Anthropic on Vertex models.
#574 opened Sep 30, 2024 by simonbutt
1
[feature request] Support the -F flag for inspect eval
#551 opened Sep 27, 2024 by tw-aisi
1
[feature request] support passing -H / --host to docker sandbox
#540 opened Sep 27, 2024 by JasonGross
1
[feature request] Support video content as input and video-LLM models
#508 opened Sep 25, 2024 by alexandraabbas
1
ProTip! Mix and match filters to narrow down what you’re looking for.