feat(bedrock): add multimodal support for documents, images and videos #1888
Conversation
- Add BedrockMediaFormat class to handle media format conversions for documents, images and videos - Enhance Media class with builder pattern and comprehensive format constants - Refactor BedrockProxyChatModel to support multimodal content handling - Add integration tests for PDF, image and video processing - Add unit tests for Media and BedrockMediaFormat classes - Upgrade AWS SDK version from 2.26.7 to 2.29.29 - Remove redundant aws.sdk.version property in favor of awssdk.version Resolves spring-projects#1887
- Added multimodal support documentation (images, video, documents) - Added deprecation notices for existing Bedrock model implementations - Updated feature comparison table - Added warning notes about transitioning to Converse API
| public Builder data(URL url) { | ||
| Assert.notNull(url, "URL must not be null"); | ||
| this.data = url.toString(); | ||
| ; |
| */ | ||
| @Deprecated |
| } | ||
|
|
||
| public Media build() { | ||
| return new Media(mimeType, data, id, name); |
There was a problem hiding this comment.
should verify that required fields are set, I presume mimeType and media are required but there can be defaults generated for id and name. The default for no media name shouldn't be nope and it seems like id could be null? It is retrieved from the model response for the Audio response.
| * @author Christian Tzolov | ||
| * @since 1.0.0 | ||
| */ | ||
| public class BedrockMediaFormat { |
| * @author Christian Tzolov | ||
| */ | ||
| @SpringBootTest(classes = BedrockNovaChatClientIT.Config.class) | ||
| @EnabledIfEnvironmentVariable(named = "AWS_ACCESS_KEY_ID", matches = ".*") |
There was a problem hiding this comment.
Can use @RequiresAwsCredentials meta annotation instead
|
|
||
| throw new IllegalArgumentException("Unknown location: " + request.location()); | ||
| }) | ||
| .inputType(WeatherRequest.class) |
There was a problem hiding this comment.
I added .description("Get the weather for a city in Celsius")
note, that the description we normally use for this didn't work .description("Get the weather in location")
| @@ -354,6 +361,91 @@ else if (prompt.getOptions() instanceof ChatOptions) { | |||
| .build(); | |||
| } | |||
|
|
|||
| private ContentBlock mapMediaToContentBlock(Media media) { | |||
|
|
|||
| var mimeType = media.getMimeType(); | |||
There was a problem hiding this comment.
should look to break up this multi-screen function into if/else structure with additional helper methods.
|
Did minor cleanup. I ran all the aws tests, made some small changes to have the tests be less brittle. getWeather function description that used to work, isn't working any more on nova. |
|
so cool to see this. merged in 51261e4 but may not get a green build in CI as the there is something going on the the Ollama tests. |
|
I'm re-opening because I don't see the code merged in main |
|
now really merged! 9a5d61c Made only |
Resolves #1887