Implement support for concatSupported = false for OpenAI models #7299
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
|
Currently we are trying to figure out where to override concatSupported to false for the OpenAI module |
|
Thanks for this - will think through how best to fix this |
|
Is there any way to help this forward, @jacoblee93? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When streaming structured output with the ChatOpenAI model the event on_chain_end is incorrect. All of the variables within data.output are a concatenation of all the intermediary streamed output messages instead of just the final result.
This happens since when combining the streamed chunks in RunnableSequence._streamIterator() we check if concatenation is supported by checking concatSupported, but this variable is hardcoded to true and the false branch of the if statement is not implemented so for every model it assumes that concatenation is supported which is not the case for the ChatOpenAI model.
To solve this issue we have implemented the false branch which takes the last streamed chunk as the final output, instead of concatenating them all together to support models which don’t support concatenation.
Fixes #7114