[Question] Custom metric type

[us2547]

Hi,
Is there a way to define custom metric that would output dictionary or tuple?
I need to create a metric that will output Scorer values aggregated by the value and display occurrence of each value.
The values in my case are strings.
It seems that the only output type from the "metric" that is supported is "float".
Here is example "metric" which is currently not supported:

@metric
def item_count() -> Metric:
    """
    Currently not working. Inspect expects a float value for the metric.
    """
    def metric(scores: list[Score]) -> tuple:
        count_dict = {}
        for score in scores:
            score_value = str(score)
            if score_value in count_dict:
                count_dict[score_value] += 1
            else:
                count_dict[score_value] = 1
        return tuple(count_dict.items())
    return metric

Thank you.

[jjallaire]

cc @dragonstyle

[dragonstyle]

What version of Inspect are you using? Metrics in recent versions of Inspect are allowed to return any of the following types:

str | int | float | bool
list[str | int | float | bool]
dict[str, str | int | float | bool | None]

The elements of the system downstream should automatically handle metrics that return lists or dicts, hopefully behaving as you'd expect them to.

[us2547]

I'm using latest inspect, version: inspect_ai==0.3.42. I have custom scorer that returns following class (example):

       # Debug
        scorer_value = "test-value"
        return_score = Score(
                value=scorer_value,
                answer=state.output.completion,
                explanation=scorer_explanation,
                metadata=metadata,
            )
        return return_score

The "value" is properly reflected as "test-value" in the json and in inspect view, however if I use custom metric, by the time Scorer.value reaches it, the value is converted to float and is showing as 0.0. Is there a way to send string values to custom metric for processing?

[dragonstyle]

I can confirm that I'm seeing the same issue (the score arrives with a float value in the metric). I am investigating now, but I suspect the issue is related to how we've implemented sample reducing (and it not dealing very nicely with strings, by default). I'll respond here once I've buttoned this down - sorry for the issue and thanks for reporting it.

[dragonstyle]

I'm reverting the original fix to this as it caused additional regressions elsewhere. I will attempt to fix again today.

[dragonstyle]

We ended up reverting the second fix to this issue as well. I will take another crack at it soon!