Feedback feature on version 1.0.1 seems not support with Claude 3

[meenonboard]

Description

I've integrated Trulens 1.0.0 to a new Python-based project , which working within AWS environment.
Follow the instruction from a quick start page with few modification to run with AWS Bedrock (the quick start provides OpenAI example).

At the run time , it seems the feedback feature for Groundedness and Answer Relevant cannot call Claude3 for generating feedback and cost tracking.

What should I do next to solve these ?

Available Model ID at https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html

Python Code

Runtime Result

(.venv) AYMAC:evaluation [REDACTED]$ python groundedness.py
🦑 TruSession initialized with db url sqlite:///default.sqlite .
🛑 Secret keys may be written to the database. See the database_redact_keys option of TruSession to prevent this.
✅ In Groundedness, input source will be set to record.app.retrieve.rets.collect() .
✅ In Groundedness, input statement will be set to record.main_output or Select.RecordOutput .
✅ In Answer Relevance, input prompt will be set to record.main_input or Select.RecordInput .
✅ In Answer Relevance, input response will be set to record.main_output or Select.RecordOutput .
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=3.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=2.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=2.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=1.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=1.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=0.
Feedback Function exception caught: Traceback (most recent call last):
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 895, in run
mod_base_endpoint.Endpoint.track_all_costs_tally(
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 524, in track_all_costs_tally
result, cbs = Endpoint.track_all_costs(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 504, in track_all_costs
return Endpoint._track_costs(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 601, in _track_costs
result: T = __func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 565, in relevance_with_cot_reasons
return self.generate_score_and_reasons(system_prompt, user_prompt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/providers/bedrock/provider.py", line 249, in generate_score_and_reasons
response = self.endpoint.run_in_pace(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 323, in run_in_pace
raise RuntimeError(
RuntimeError: Endpoint bedrock request failed 4 time(s):
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 910, in run
raise RuntimeError(
RuntimeError: Evaluation of Answer Relevance failed on inputs:
{'prompt': 'new car on top incentive for high ticket size',
'response': '\n'
"['# Incentive Program for S.
bedrock request failed <class 'botocore.errorfactory.ValidationException'>=An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Retries remaining=0.
Error removing trivial statements: Endpoint bedrock request failed 4 time(s):
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.
An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-haiku-20240307" is not supported on this API. Please use the Messages API instead.. Proceeding with all statements.
Feedback Function exception caught: Traceback (most recent call last):
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 895, in run
mod_base_endpoint.Endpoint.track_all_costs_tally(
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 524, in track_all_costs_tally
result, cbs = Endpoint.track_all_costs(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 504, in track_all_costs
return Endpoint._track_costs(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/endpoint.py", line 601, in _track_costs
result: T = __func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 1434, in groundedness_measure_with_cot_reasons
futures = [
^
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/feedback/llm_provider.py", line 1435, in
executor.submit(evaluate_hypothesis, i, hypothesis)
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/utils/threading.py", line 78, in submit
return super().submit(
^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/[REDACTED]/gami-repo/simple-rag/.venv/lib/python3.11/site-packages/trulens/core/feedback/feedback.py", line 910, in run
raise RuntimeError(
RuntimeError: Evaluation of Groundedness failed on inputs:
{'source': [['# Incentive Program for Sales B2B Nationwide (FTE/OA)\n'
'## 1 Volume and Loss Management Incentive\n.

Environment

OS: MacOS
Python Version: 3.11.9
TruLens version: 1.0.0
Platform : AWS Bedrock
LLM version : anthropic.claude-3-haiku-20240307-v1:0

[sfc-gh-jreini]

Hey @meenonboard - Claude 3 support in this PR will be available in the next release. Closing this as it's duplicate of your other discussion.

[meenonboard]

Hey @meenonboard - Claude 3 support in this PR will be available in the next release. Closing this as it's duplicate of your other discussion.

Thank you for the update on this change.