Skip to content

Commit

Permalink
Support "omni" as model name
Browse files Browse the repository at this point in the history
  • Loading branch information
thekid committed Oct 13, 2024
1 parent 9eaaeda commit a402fdf
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 7 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,18 @@ This library implements OpenAI APIs.

TikToken
--------
Encodes text to tokens. Download the [cl100k_base](https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken) and [o200k_base](https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken) vocabularies first!
Encodes text to tokens. Download the vocabularies [cl100k_base](https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken) (used for GPT-3.5 and GPT-4.0) and [o200k_base](https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken) (used for Omni and O1) first!

```php
use com\openai\{Encoding, TikTokenFilesIn};

$source= new TikTokenFilesIn('.');

// GPT 3.5, 4.0 => [9906, 4435, 0]
// By name => [9906, 4435, 0]
$tokens= Encoding::named('cl100k_base')->load($source)->encode('Hello World!');

// GPT o1, omni => [13225, 5922, 0]
$tokens= Encoding::named('o200k_base')->load($source)->encode('Hello World!');
// By model => [13225, 5922, 0]
$tokens= Encoding::for('omni')->load($source)->encode('Hello World!');
```

See also
Expand Down
7 changes: 4 additions & 3 deletions src/main/php/com/openai/Encoding.class.php
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,10 @@ public static function named(string $name): self {
*/
public static function for(string $model): self {
static $models= [
'/^o1/' => 'o200k_base',
'/^gpt-4o/' => 'o200k_base',
'/^gpt-4/' => 'cl100k_base',
'/^o1/' => 'o200k_base',
'/^omni/' => 'o200k_base',
'/^gpt-4o/' => 'o200k_base',
'/^gpt-4/' => 'cl100k_base',
'/^gpt-3.?5/' => 'cl100k_base',
];

Expand Down

0 comments on commit a402fdf

Please sign in to comment.