The system is happy about the model saying it doesn't know something even though it has been stated in the prompt

[loulblemo]

from trulens_eval import OpenAI as fOpenAI
provider = fOpenAI()

from trulens_eval import Feedback

f_qa_relevance = Feedback(
    provider.relevance_with_cot_reasons,
    name="Answer Relevance"
).on_input_output()

f_qa_relevance("My favorite color is blue, what is my favorite color?", "I don't know")

(1.0,
 {'reason': 'Criteria: The response directly addresses the prompt by stating "I don\'t know."\nSupporting Evidence: The response provided is a direct answer to the prompt, acknowledging that the responder does not know the favorite color of the user. This aligns perfectly with the criteria of the evaluation, as it is a relevant and intentional refusal to answer the question posed in the prompt. Therefore, the response deserves a high score of 10 for being the most relevant in this context.'})

[sfc-gh-jreini]

Hey @loulblemo - this is intended. Abstentions (LLM saying when it doesn't know) are a desired quality. That being said, we're thinking of ways to better handle this case specifically - notably for groundedness evaluations.

[sfc-gh-jreini]

Hey @loulblemo - wanted to let you know we've updated this treatment.

For answer relevance, abstentions will now be counted as not relevant.
For groundedness, abstentions will be counted as grounded.

Additionally, you can also use a special version of groundedness that assesses answerability. In this case, answerable abstentions will be considered not grounded (score 0), and unanswerable abstentions will still be grounded (score 1).