Assistance Needed with Custom Composite Feedback Function Using Groundedness and Answer Relevance

[ciliamadani]

What I’m Trying to Achieve:
I am working on creating a custom feedback function in TruLens that combines the results of groundedness and answer relevance. The goal is to calculate a composite score by weighting the outputs of these two feedback mechanisms. However, I am unsure if my approach is correct, especially regarding how I’m using f_groundedness.run() and f_answer_relevance.run() within my function.

My Current Implementation:
Below is my implementation for context:

from trulens.core import Feedback, Select, TruSession, FeedbackMode
from trulens.providers.openai import OpenAI
from dotenv import load_dotenv

# Initialize the OpenAI provider
provider = OpenAI(model_engine="gpt-4o-mini")

# Define weights for the composite score
weight_groundedness = 0.8
weight_answer_relevance = 0.1
weight_context_relevance = 0.1

# Define groundedness feedback function
f_groundedness = (
    Feedback(
        provider.groundedness_measure_with_cot_reasons, name="Groundedness"
    )
    .on(Select.RecordCalls.retrieve.rets)
    .on_output()
)

# Define answer relevance feedback function
f_answer_relevance = (
    Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance")
    .on_input()
    .on(Select.RecordCalls.query.rets)
)

# Composite feedback function
def composite_feedback_function(record):
    print(record)  # Debugging the record structure

    # Retrieve individual feedback scores
    groundedness_score = f_groundedness.run(record=record).score
    answer_relevance_score = f_answer_relevance.run(record=record).score

    # Weighted average of scores
    composite_score = (
        weight_groundedness * groundedness_score +
        weight_answer_relevance * answer_relevance_score
    )
    return composite_score

# Wrap the composite function in a Feedback object
f_composite = Feedback(
    imp=composite_feedback_function, name="Composite Feedback"
).on(Select.RecordCalls)

My Concerns:
Correctness of f_groundedness.run() and f_answer_relevance.run():
Am I using .run() correctly to evaluate the feedback scores within the custom composite function?

Record Structure:
I am debugging the record structure with print(record) but not sure if this is the right way to ensure compatibility.

Composite Feedback Initialization:
Is wrapping the composite function in a Feedback object as shown above appropriate, or am I missing something in the process?

[dosubot[bot]]

To address your concerns about the implementation of your custom composite feedback function in TruLens, let's go through each point:

1. Correctness of f_groundedness.run() and f_answer_relevance.run():

   • Your use of the .run() method to evaluate feedback scores within the custom composite function seems correct. The run method is designed to execute the feedback function on a given record and return a FeedbackResult object, which includes the score you are accessing ^[1][2].

2. Record Structure Debugging:

   • Printing the record is a good initial step to understand its structure. However, ensure that the record parameter you pass to the run method is an instance of record_schema.Record, as this is required for compatibility ^[2]. If you encounter issues, you might want to inspect specific fields within the record to ensure they match the expected input for your feedback functions.

3. Composite Feedback Initialization:

   • Wrapping your composite function in a Feedback object as shown is appropriate. This allows you to manage the execution and evaluation of your composite feedback function within the TruLens framework. Ensure that the Select.RecordCalls is correctly referencing the parts of the record you intend to use ^[2].

Overall, your approach seems well-structured. If you encounter specific issues, consider checking the compatibility of the record structure with the expected inputs of your feedback functions and ensure that the Select statements correctly target the data you wish to evaluate.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}