Implement realtime API

[thekid]

This pull request implements #8 in a new com.openai.realtime package. Quoting the OpenAI platform docs:

The Realtime API enables you to build low-latency, multi-modal conversational experiences. It currently supports text and audio as both input and output, as well as function calling.

The Azure API also supports images by passing ['type' => 'ms_image', 'image' => 'data:image/webp;base64,...'], which is not documented but can be reversed engineered from the API's error messages.

Example

This example uses Azure OpenAI version:

use com\openai\realtime\RealtimeApi;
use util\cmd\Console;

$api= new RealtimeApi('wss://example.openai.azure.com/openai/realtime?'.
  '?api-version=2024-10-01-preview'.
  '&deployment=gpt-4o-realtime-preview'
);
$session= $api->connect(['api-key' => getenv('AZUREAI_API_KEY')]);
Console::writeLine($session);

// Send prompt
$api->transmit([
  'type' => 'conversation.item.create',
  'item' => [
    'type'    => 'message',
    'role'    => 'user',
    'content' => $content,
  ]
]);

// Receive response(s)
$api->send(['type' => 'response.create', 'response' => ['modalities' => ['text']]]);
do {
  $event= $api->receive();
  Console::writeLine($event);
} while ('response.done' !== $event['type'] && 'error' !== $event['type']);

$api->close();

See also

• https://openai.com/index/introducing-the-realtime-api/
• https://platform.openai.com/docs/guides/realtime
• https://github.com/openai/openai-realtime-api-beta/
• https://github.com/azure-samples/aoai-realtime-audio-sdk

[thekid]

The structure for the tools element differs from that used in the REST API in that it has no function substructure:

Realtime

[
  [
    'type'        => 'function',
    'name'        => 'get_person',
    'description' => 'Gets a person',
    'parameters'  => [...]
  ]
]

REST

[
  [
    'type'        => 'function',
    'function'    => [
      'name'        => 'get_person',
      'description' => 'Gets a person',
      'parameters'  => [...]
    ]
  ]
]

This means the com.openai.Tools class cannot completely be used in a generic way, and we might need to specialize it for the two APIs.

[thekid]

$api->connect(['api-key' => getenv('AZUREAI_API_KEY')]);

// Initial handshake: Server sends session.created
Console::writeLine($api->receive());

Should be return the session.created event from the connect() call? The documentation states this event is sent as soon as the connection is successfully established. Provides a connection-specific ID that may be useful for debugging or logging (source).

[thekid]

Integration "test": https://gist.github.com/thekid/678620dfb548e6a972cb268c8b5b7899

[thekid]

This means the com.openai.Tools class cannot completely be used in a generic way, and we might need to specialize it for the two APIs.

...or we make use of something like xp-forge/marshalling#7