[Bug] Evaluator not recieving correct_answer on run

[d-pollard]

Describe the bug

Whenever you run an evaluator on a testset, the saved correct_answer is not retained; this leads to the error: Missing input: At 'correct_answer' - it also clears out the correct_answer field when it is ran from the evaluator screen

Setup

Using cloud version

To Reproduce

1. Create a prompt and setup JSON output
2. Create a testset
3. Add correct_answer
4. Create evaluator using JSON Multi-Field Match
5. Run testset from evaluator screen
6. Error

Expected behavior

Just like an evaluation, I would expect the evaluator run page to retain correct_answer

Screenshots

No response

Agenta SDK Version

No response

Agenta Image Tag

No response

Important Context

LinkedIn details

No response

[linear-code]

AGE-3833

[unfitcoder101]

Hi @d-pollard @mmabrouk - I'd like to investigate this. Quick clarification before I dive in:

1. Is the bug on the frontend (correct_answer being stripped before sending to evaluator API) or backend (API not forwarding it to the evaluator service)?
2. Should I focus on the evaluator run screen specifically, or also the broader evaluation flow?

Will start tracing through the code now from EvaluatorRun components. Happy to assign this to myself if that works for you.

[mmabrouk]

@unfitcoder101 it would be awesome if you could record a video reproducing the issue and check the network tab (jam.dev could be helpful here). Or @d-pollard if available. It would help us understand what is happening.

Also, this PR might have fixed the issue (although. It's coming to 103.5.

[unfitcoder101]

@mmabrouk reproduced the bug. Here's what I found:
When loading a testset from the Evaluator screen, correct_answer column gets silently hidden. The UI shows: "1 unused testcase column hidden because the prompt does not reference them"
This means the evaluator runs without access to correct_answer, causing Contains JSON → Success: false even on valid outputs.
Steps to reproduce:

Evaluators → Contains JSON → Select app → Connect test set
Load a testset that has country + correct_answer columns
correct_answer is immediately hidden by the UI
Run all → Contains JSON Success: false

Root cause hypothesis: The testset loader filters columns not referenced in the prompt template. correct_answer should always pass through to the evaluator regardless of prompt references.
I have screenshots showing the hidden column + failed eval result. Happy to dig into the fix — should I look at the testset loader component on the frontend?

[Dewnethmi]

Hi! I'm interested in contributing to Agenta and would love to join the community Slack. The shared invite link on the website appears to have expired. Could someone share an updated link? Thanks!

[unfitcoder101]

@mmabrouk checked PR #4701 — can confirm whether this fixes it once it lands
in the cloud version. In the meantime, I've also built a RAG chatbot example
using Vercel AI SDK + Agenta tracing, live at agenta-ai-examples.vercel.app
— ready whenever you want to review it for the docs/examples repo.