Req Interceptor with audio files

[fragakos]

I am using request Inteseptor to catch an audio file, transcribe it and return as a text. I know the details when text is sent are different that when a audio file is sent. When a audio file is detected the returned vaulue will be an RequestDetails object constructed by me. But the OpenAi stream api received and [object Object] and not the json it is expecting.
Any ideas??

This is what i am returning from the interceptor

[OvidijusParsiunas]

Hi @fragakos.
First question, which OpenAI API are you calling? Is it the default OpenAI Chat Completion API with Streaming?
If it is, then your outgoing body is missing the stream: true property and your headers are missing the Authorization property that contains your OpenAI API key.

This is showcased in an example in the link above:

I would also recommend checking what Deep Chat passes in the details argument of the requestInterceptor and once you modify or create your own response try to make sure it uses a similar structure to it.

So to add the Authorization header all you need to do is pass the headers object in the returned object, e.g:

chatElementRef.requestInterceptor = (details) => {
return {
  body: {
    model: 'gpt-4o',
      ...
    },
    headers: {
      Authorization: 'Bearer your-key',
      'Content-Type': 'application/json',
    },
  };
};

Let me know if this helps you.

[OvidijusParsiunas]

The functionality you are describing seems very familiar to the new OpenAI Realtime API.
We are going to start the development for its integration in Deep Chat very soon!

[fragakos]

I am using the stream open ai api but I've set up a dummy response in order to do my testing. The bug occurs before the Everything in the api.
I used a template deep chat for stream responses and I added the microphone option. When a text is sent, the interceptor does nothing, and everything works. When an audio file is sent, I catch it, transcribe it and try to return the details like it was a text in the first place.
The details of when a text is sent and when I construct the new details of an audio file, are exactly the same. Still the api receives a object which is not a valid json

[OvidijusParsiunas]

That is strange, I would like to see your configuration for Deep Chat and how you have set up your code. Thanks.

[fragakos]

This is the deep chat component, I am using next btw
<DeepChat style={{ borderRadius: "10px" }} introMessage={{ text: "Send a streamed chat message through an example server to OpenAI.", }} connect={{ url: "/api/openai/chat-stream", stream: true, additionalBodyProps: { model: "gpt-4o" }, }} requestInterceptor={handleRequestIntercept} responseInterceptor={handleRespondIntercept} microphone={true} requestBodyLimits={{ maxMessages: -1 }} errorMessages={{ displayServiceErrorMessages: true }} validateInput={(text?: string, files?: File[]) => { return ( (!!text && text.trim() !== "" && (!files || files.length === 0)) || ((!text || text.trim() === "") && !!files && files.length > 0) ); }} />
I print the details before returning,
These are the untouched ones

An these are the constructed ones

Here is the code for the interceptor
`import { processFile } from "../utils/requestProcessor";
import { extractFileFromRequestBody } from "../utils/fileUtils";

export const handleRequestIntercept = (details: any) => {
console.log("Request intercepted:", details);
const STORAGE_KEY = "chat_history";
// return details;
try {
const file = extractFileFromRequestBody(details.body);
const savedDetails = JSON.parse(
localStorage.getItem(STORAGE_KEY) || JSON.stringify(details)
);

if (file) {
  const response = processFile(file, savedDetails);
  localStorage.setItem(STORAGE_KEY, JSON.stringify(response));
  details.body = response.body;
  details.headers = response.headers;
  console.log("AUDIO RESPONSE SAVED", details);
  return details;
} else {
  const existingHistory = JSON.parse(
    localStorage.getItem(STORAGE_KEY) || JSON.stringify(details)
  );
  if (
    details.body.messages.length !== existingHistory.body.messages.length
  ) {
    existingHistory.body.messages = [
      ...existingHistory.body.messages,
      details.body.messages[details.body.messages.length - 1],
    ];
  }

  // Save updated chat history
  localStorage.setItem(STORAGE_KEY, JSON.stringify(existingHistory));
  console.log("CHAT HISTORY SAVED", details);
  details.headers = { "content-type": "application/json" };
  return details;
}

} catch (error: any) {
return {
error: Error processing request: ${error.message},
};
}
};
`

[OvidijusParsiunas]

Hi @fragakos.
Today (Japan Time) I have very little free time so it is hard to pinpoint the exact cause of the issue without properly examining it. I will take a look at it tomorrow.

May I ask, in the two given images, is the second one causing an error?
May I ask what the object that you are sending looks like when you receive an error?
Also, if you are using NextJs, there could be a problem in the backend. Especially with the object that is going to the OpenAI API and not the NextJs server itself.

Let me know if you have any more info on this. Thankyou!

[fragakos]

The second image is the problem, even though they look the same.
The object never reaches open ai api, it gives an error in the line const textRequestBody = (await req.json()) as DeepChatOpenAITextRequestBody; with this error "Error parsing request body: [SyntaxError: "[object Object]" is not valid JSON]"

[OvidijusParsiunas]

This is very interesting!! So the issue is in the frontend and more specifically in Deep Chat itself then??

[OvidijusParsiunas]

Just to save me time trying to reproduce the error tomorrow, I would really appreciate if you could give me the minimal configuration required to reproduce the error. 🙇
Thankyou!

[fragakos]

What do you mean with minimal configuration?

[OvidijusParsiunas]

Hey @fragakos.

I have looked at your code and everything appears to be in order. I am very surprised that you are getting that error.
If the JSONs that you are sending to the /chat-stream API are exactly what is in your images, then everything should be working.

I am wondering if there some sort of middleware that you are using in your NextJS project or perhaps you are using other packages to that work with http? If everything is the same as the examples provided, then I am unfortunately not sure why you are getting that error. 😢

I would suggest inspecting the actual outgoing request in your browser's console and see if the body/headers are correct. If they are, then maybe the issue is on the actual backend.

Apologies that I could not help you more. 🙇
If you do solve the problem please update this thread so other developers can use it to troubleshoot their issues.

Thankyou!