Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Count tokens with tiktoken #4704

Merged
merged 3 commits into from
Jun 15, 2023
Merged

Count tokens with tiktoken #4704

merged 3 commits into from
Jun 15, 2023

Conversation

waynehamadi
Copy link
Contributor

@waynehamadi waynehamadi commented Jun 15, 2023 •
edited
Loading

Background

we route each models to a tiktoken encoding model

Changes

Documentation

Test Plan

PR Quality Checklist

  • My pull request is atomic and focuses on a single change.
  • I have thoroughly tested my changes with multiple different prompts.
  • I have considered potential risks and mitigations for my changes.
  • I have documented my changes clearly and comprehensively.
  • I have not snuck in any "extra" small tweaks changes.
  • I have run the following commands against my code to ensure it passes our linters: 
    black .
isort .
mypy
autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring autogpt tests --in-place

@waynehamadi waynehamadi force-pushed the count_tokens_with_tiktoken branch from 6b51755 to 082b154 Compare June 15, 2023 15:35
@github-actions github-actions bot added size/m and removed size/l labels Jun 15, 2023
@waynehamadi waynehamadi force-pushed the count_tokens_with_tiktoken branch 4 times, most recently from be03bcc to 9ed71b3 Compare June 15, 2023 15:59
@github-actions github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Jun 15, 2023
@github-actions
Copy link
Contributor

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

@github-actions github-actions bot added size/l and removed size/m labels Jun 15, 2023
@erik-megarad @waynehamadi
Update OpenAI model info and remove duplicate modelsinfo.py (Signific…
a1d115c 
…ant-Gravitas#4700)

* Update OpenAI model info and remove duplicate modelsinfo.py

* Fix max_tokens for gpt-4-0613

Signed-off-by: Merwane Hamadi <[email protected]>
Co-authored-by: Merwane Hamadi <[email protected]>
@waynehamadi waynehamadi force-pushed the count_tokens_with_tiktoken branch from 9ed71b3 to f5f2b24 Compare June 15, 2023 16:01
@github-actions github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Jun 15, 2023
@github-actions
Copy link
Contributor

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

@github-actions github-actions bot added size/m and removed size/l labels Jun 15, 2023
@erik-megarad @waynehamadi
Update count_message_tokens to support new OpenAI models
96f5373 
Signed-off-by: Merwane Hamadi <[email protected]>
Co-authored-by: Merwane Hamadi <[email protected]>
@waynehamadi waynehamadi force-pushed the count_tokens_with_tiktoken branch from f5f2b24 to 96f5373 Compare June 15, 2023 16:03
@waynehamadi waynehamadi merged commit 3525a4b into Significant-Gravitas:Release-v0.4.1 Jun 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Pwuts Pwuts Pwuts approved these changes

Assignees
No one assigned
Labels
size/m
Projects
AutoGPT development kanban
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@waynehamadi @Pwuts @erik-megarad