When I send user feedback on a trace through the API and then open the Observability filter, I expect to filter traces by that feedback (for example by score or comment). The feedback fields never show up in the filter, so there is no way to filter by them. This happens for every evaluator's feedback, not just API-created ones.
Steps to reproduce
- Send feedback on a trace:
curl -X POST "$HOST/api/simple/traces/" \
-H "Authorization: ApiKey $KEY" -H "Content-Type: application/json" \
-d '{"trace":{"data":{"outputs":{"score":1,"comment":"looks good"}},
"references":{"evaluator":{"slug":"user-feedback"}},
"links":{"invocation":{"trace_id":"<32hex>","span_id":"<16hex>"}}}}'
- Open Observability, open Filter, add an "Is Annotation" condition, pick the
user-feedback evaluator, then open the Feedback field dropdown.
- Expected:
score and comment are selectable feedback fields. Actual: the dropdown is empty.
Debugging info
The evaluator is created, and its stored output schema (from POST /api/preview/simple/evaluators/query) nests the fields under an extra outputs key:
{
"type": "object",
"properties": {
"outputs": {
"type": "object",
"properties": { "score": {"type": "integer"}, "comment": {"type": "string"} }
}
}
}
To do
When I send user feedback on a trace through the API and then open the Observability filter, I expect to filter traces by that feedback (for example by
scoreorcomment). The feedback fields never show up in the filter, so there is no way to filter by them. This happens for every evaluator's feedback, not just API-created ones.Steps to reproduce
user-feedbackevaluator, then open the Feedback field dropdown.scoreandcommentare selectable feedback fields. Actual: the dropdown is empty.Debugging info
The evaluator is created, and its stored output schema (from
POST /api/preview/simple/evaluators/query) nests the fields under an extraoutputskey:{ "type": "object", "properties": { "outputs": { "type": "object", "properties": { "score": {"type": "integer"}, "comment": {"type": "string"} } } } }To do