evals4j

Open-source evaluators for LLM applications on the JVM, for Spring AI and LangChain4j.

A Java port of LangChain's OpenEvals — the same LLM-as-judge engine, the same 33-prompt catalog, the same trajectory and structured-output matchers. evals4j is an independent project and is not affiliated with or endorsed by LangChain, Inc.

Evals are to LLM applications what tests are to ordinary software: a way to find out whether a prompt change made things better or worse before your users do.

LlmAsJudge conciseness = LlmAsJudge.builder()
        .prompt(Prompts.CONCISENESS_PROMPT)
        .feedbackKey("conciseness")
        .model(judge)
        .build();

EvaluatorResult result = conciseness.evaluate(
        "How is the weather in San Francisco?",
        "Thanks for asking! The current weather in San Francisco is sunny and 90 degrees.");

result.score();    // false
result.comment();  // "The response includes an unnecessary pleasantry ... Thus, the score should be: false."

---

Install

Requires Java 17+. Add the BOM, then whichever modules you need.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.dvarahq.oss</groupId>
      <artifactId>evals4j-bom</artifactId>
      <version>0.4.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

module	what it gives you
evals4j-core	every evaluator, the prompt catalog, the SPI. No AI-framework dependency.
evals4j-springai	drives evaluators with a Spring AI ChatModel
evals4j-langchain4j	drives evaluators with a LangChain4j ChatModel
evals4j-spring-boot-starter	auto-configures whichever of the two is on your classpath
evals4j-sandbox	Docker and local sandbox runners, for evaluating generated code
evals4j-junit5	assertions, and @EvalSuite to report the trend across runs

Runnable examples live in evals4j-examples — start with AgentTrajectoryExample, which needs no API key.

Spring Boot

Add the starter and you are done — an existing ChatModel bean is enough.

<dependency>
  <groupId>com.dvarahq.oss</groupId>
  <artifactId>evals4j-spring-boot-starter</artifactId>
</dependency>
<dependency>
  <groupId>com.dvarahq.oss</groupId>
  <artifactId>evals4j-springai</artifactId>
</dependency>

@Autowired JudgeModel judge;   // auto-configured from your ChatModel

Without Spring Boot

// Spring AI
JudgeModel judge = SpringAiJudgeModel.of(chatModel);

// LangChain4j
JudgeModel judge = LangChain4jJudgeModel.of(chatModel);

Scoring a real agent run

To judge the path an agent actually took, convert its conversation and hand it to a trajectory evaluator. Tool calls keep their ids, names and arguments, which is what the matchers compare on.

List<ChatMessage> trajectory = SpringAiMessages.fromSpringAi(messages);
// or LangChain4jMessages.fromLangChain4j(chatMemory.messages());

evaluator.evaluate(EvalRequest.of(null, trajectory, referenceTrajectory));

---

LLM-as-judge

The centrepiece. A model grades an output against a prompt, and returns a structured score plus its reasoning.

LlmAsJudge judge = LlmAsJudge.builder()
        .prompt(Prompts.CORRECTNESS_PROMPT)
        .feedbackKey("correctness")
        .model(judgeModel)
        .build();

EvaluatorResult result = judge.evaluate(
        "Who was the first president of the United States?",
        "John Adams",
        "George Washington");   // reference

Score shapes. Boolean by default; .continuous(true) for 0.0–1.0; .choices(0.0, 0.5, 1.0) to restrict it to specific values. Choices win over continuous.

Reasoning is on by default. It costs tokens, but it is what makes a low score actionable — and the schema forces the model to reason before it scores, which measurably improves consistency. Turn it off with .useReasoning(false).

Extra prompt variables are supplied per call:

judge.evaluate(EvalRequest.builder()
        .outputs(answer)
        .variable("context", retrievedDocuments)   // fills {context}
        .build());

Few-shot examples calibrate the judge:

.fewShotExample(FewShotExample.of("2+2?", "4", true, "correct and direct"))
.fewShotExample(FewShotExample.of("2+2?", "Well, mathematically...", false, "padded"))

Custom output schema for classification rather than scoring — read it with evaluateRaw:

JsonNode raw = LlmAsJudge.builder()
        .prompt("Classify: {outputs}")
        .model(judgeModel)
        .outputSchema(mySchema)
        .build()
        .evaluateRaw(EvalRequest.ofOutputs(ticket));

---

Prompt catalog

33 prompts, copied verbatim from OpenEvals and verified byte-for-byte by a checksum test.

category	prompts
quality	CORRECTNESS, CONCISENESS, HALLUCINATION, ANSWER_RELEVANCE, CODE_CORRECTNESS (+ _WITH_REFERENCE_OUTPUTS), PLAN_ADHERENCE, LAZINESS
rag	RAG_GROUNDEDNESS, RAG_HELPFULNESS, RAG_RETRIEVAL_RELEVANCE
safety	TOXICITY, FAIRNESS
security	PII_LEAKAGE, PROMPT_INJECTION, CODE_INJECTION
trajectory	TRAJECTORY_ACCURACY (+ _WITH_REFERENCE), TOOL_SELECTION
conversation	PERCEIVED_ERROR, WINS, TASK_COMPLETION, KNOWLEDGE_RETENTION, USER_SATISFACTION, AGENT_TONE, LANGUAGE_DETECTION, SUPPORT_INTENT
image	EXPLICIT_CONTENT, SENSITIVE_IMAGERY
voice	AUDIO_QUALITY, TRANSCRIPTION_ACCURACY, USER_INTERRUPTS, VOCAL_AFFECT

Each constant's Javadoc lists the variables it needs. Notably: the RAG prompts want {context}, PLAN_ADHERENCE wants {plan}, the injection prompts grade {inputs} only, and the conversation and trajectory prompts take a whole message list as outputs.

Which RAG evaluator?

evaluator	compares	needs a reference?
correctness	answer vs. expected answer	yes
helpfulness	answer vs. question	no
groundedness	answer vs. retrieved context	no
retrieval relevance	retrieved context vs. question	no

Multimodal

Prompts with an {attachments} placeholder take images or audio, spliced in as content blocks:

judge.evaluate(EvalRequest.builder()
        .inputs("describe this")
        .outputs(caption)
        .attachment(Attachment.ofUrl("https://example.com/photo.png"))
        .build());

---

Evaluators that need no model

ExactMatch.create()               // canonical JSON, map key order ignored
LevenshteinDistance.create()      // normalized similarity in [0,1]
EmbeddingSimilarity.create(embeddings)   // cosine (default) or dot product

Structured output

Compare extracted objects key by key. Keys with a rubric are judged by a model; the rest are compared for equality — which is exactly what you want when one field is a free-text summary and the rest are exact values.

Evaluator evaluator = JsonMatchEvaluator.builder()
        .excludeKeys("id", "extracted_at")
        .rubric("summary", "Does the summary capture the main points of the reference?")
        .model(judgeModel)
        .build();

List<EvaluatorResult> results = evaluator.evaluateAll(
        EvalRequest.of(null, extracted, expected));
// json_match:summary, json_match:customer_name, ...

Add .aggregator(AVERAGE) or .aggregator(ALL) to collapse to one score. Lists are supported with four pairing strategies (.listMatchMode(...)) and a second aggregation across elements (.listAggregator(...)).

Agent trajectories

Compare the tool calls an agent made against a reference run.

Evaluator evaluator = TrajectoryMatchEvaluator.builder()
        .matchMode(TrajectoryMatchMode.SUPERSET)          // agent did at least the reference calls
        .toolArgsMatchMode(ToolArgsMatchMode.EXACT)
        .overrideOnKeys("book_flight", "flight_no")       // ignore the timestamp on this one tool
        .build();

mode	passes when
STRICT	same messages, same roles, same tool calls (message content is never compared)
UNORDERED	same set of tool calls, any order or grouping
SUBSET	every call the agent made appears in the reference — it may do less
SUPERSET	every reference call was made — it may do more

Per-tool overrides take a match mode, a list of dotted argument paths ("time.start"), or your own predicate.

When there is no single correct path, judge it instead:

TrajectoryLlmAsJudge.builder().model(judgeModel).build()
        .evaluate(EvalRequest.ofOutputs(messages));

---

Generated code

// Does it compile? Uses the JDK Compiler API in-process — nothing to install.
JavacEvaluator.builder()
        .extractionStrategy(CodeExtractionStrategy.MARKDOWN_CODE_BLOCKS)
        .build();

// Does it run? Needs a sandbox.
ExecutionEvaluator.builder()
        .sandbox(DockerSandboxRunner.builder().image("python:3.12-slim").build())
        .fileName("outputs.py")
        .command("python", "outputs.py")
        .build();

// Does it do what was asked?
CodeLlmAsJudge.builder().model(judgeModel).build();

// Any other checker: mypy, tsc, ruff, shellcheck.
CliCheckEvaluator.builder().command("mypy", "--ignore-missing-imports").fileName("outputs.py").build();

Extraction strategies: NONE, MARKDOWN_CODE_BLOCKS (skips shell and JSON blocks), LLM, or your own function. When nothing can be extracted the result is a failure carrying code_extraction_failed metadata, not an exception.

Running model-generated code is running untrusted input. DockerSandboxRunner defaults to no network, capped memory and CPU, a read-only root filesystem, and all Linux capabilities dropped. LocalProcessSandboxRunner is not a security boundary — it is for CI and for code you wrote.

---

Multi-turn simulation

Some failures only appear over several turns. Simulate a user, let your app respond, and score the whole transcript.

SimulationResult result = MultiturnSimulation.builder()
        .app((message, threadId) -> myAgent.chat(threadId, message.text()))
        .user(SimulatedUser.llm("You are a customer whose order never arrived.", judgeModel))
        .maxTurns(5)
        .stopWhen((trajectory, turn) -> mentionsRefund(trajectory))
        .trajectoryEvaluator(LlmAsJudge.builder()
                .prompt(Prompts.TASK_COMPLETION_PROMPT)
                .feedbackKey("task_completion")
                .model(judgeModel)
                .build())
        .run();

result.trajectory();        // the full conversation
result.evaluatorResults();  // the scores

The app receives only the newest user message and a stable threadId, so the simulation exercises your app's own memory rather than replaying a transcript into it. SimulatedUser.scripted(...) takes a fixed list instead.

---

In your test suite

@Test
void answersStayConcise() {
    EvalAssert.assertPassed(conciseness.evaluate(question, myApp.answer(question)));
}

@Test
void answersStayCorrect() {
    EvalAssert.assertScoreAtLeast(0.8, correctness.evaluate(question, answer, reference));
}

Failure messages carry the judge's reasoning, because "expected true but was false" tells you nothing about an LLM score.

EvalReport implements EvalTracer; register it on your evaluators and write a summary at the end of the run — the trend across runs is what matters, not one pass or fail.

@EvalSuite does the bookkeeping: it supplies the report as a test parameter, shares one across every class in the run, and writes it out, so no suite needs an @AfterAll calling writeMarkdown.

@EvalSuite
class ConcisenessEvalTest {

    @Test
    void staysConcise() {
        LlmAsJudge judge = LlmAsJudge.builder()
                .prompt(Prompts.CONCISENESS_PROMPT)
                .model(judgeModel)
                .build();

        EvalAssert.assertPassed(judge.evaluate(question, answer));
    }
}

No .tracer(...) anywhere: the extension opens an EvalScope around each test, and an evaluator built without a tracer reports into whatever scope is open. An evaluator that was given one keeps it — configured always beats ambient — so a suite mixing the two behaves as written. Take an EvalReport parameter when you want to assert on the collected results directly.

The report goes to target/evals4j-report.md, or wherever -Devals4j.report=… says. One file covers the whole run rather than one per class, because a mean drifting from 0.9 to 0.7 over a month is the signal, and per-class files fragment it.

A .json twin is written beside it and read back next run as a baseline, which is what puts the change column in the table:

evaluator	mean score	n	change
conciseness	0.870	30	-0.030

Keep that file between runs — commit it, or cache it in CI — and the report shows movement rather than a snapshot. -Devals4j.report.maxDrop=0.05 fails the run when a key falls further than you will tolerate.

If a whole module is eval suites and even the annotation is noise, set junit.jupiter.extensions.autodetection.enabled=true and drop it; the extension registers itself through META-INF/services. JUnit leaves autodetection off by default, so nothing happens until you ask for it.

---

Running evaluations concurrently

evaluateAsync composes over the model's own asynchronous call rather than parking a blocking one on a thread, and MultiturnSimulation.runAsync scores its trajectory evaluators together. Judge calls are network calls, so give them a pool sized for waiting rather than the common pool:

ExecutorService pool = Executors.newFixedThreadPool(16);
List<CompletableFuture<EvaluatorResult>> scores = cases.stream()
        .map(c -> judge.evaluateAsync(EvalRequest.of(c.question(), c.answer()), pool))
        .toList();

The JSON-match evaluator judges each rubric key concurrently on this path, instead of one after another.

In production, an application with a Micrometer MeterRegistry gets a MicrometerEvalTracer automatically from the starter, publishing evals4j.evaluation.score and evals4j.evaluation.duration tagged by evaluator and feedback key. Boolean scores record as 0 or 1, so the mean is the pass rate.

---

Writing your own

Evaluator startsWithGreeting = Evaluator.from(
        "greeting_check", "greeting",
        request -> ScorerOutput.of(
                request.outputs().toString().startsWith("Hello"),
                "checked the opening words"));

ScorerRunner turns the output into results and fires the tracer, so a custom evaluator behaves like a built-in one.

---

Structured output and providers

The judge asks the model for JSON conforming to a schema. Providers differ in how well they support that, so both adapters take a StructuredOutputMode:

• AUTO (default) — use the provider's schema enforcement, fall back to prompting if it is rejected. Spring AI goes through StructuredOutputChatOptions; LangChain4j through JsonRawSchema, which passes the schema through untouched.
• NATIVE — require provider-side enforcement, fail if unsupported.
• PROMPTED — put the schema in the prompt and recover the JSON from the reply. Works with any model, including local ones.

Prompted-mode recovery scans for a brace-balanced region, ignoring braces inside string literals, so a fenced or chatty reply still parses.

---

Build

./mvnw verify                # unit tests, fully offline, no API key
OPENAI_API_KEY=... ./mvnw -Pit verify   # plus end-to-end tests against a real model

Sandbox tests requiring Docker skip themselves when no daemon is reachable. The examples are compiled on every build, so an API change that breaks them breaks CI.

Publishing to Maven Central is handled by the release workflow — it needs the Central and GPG secrets set on the repository.

---

Differences from OpenEvals

Documented in PARITY.md, including where a bug in upstream was fixed rather than reproduced.

License

MIT. Prompt texts and several algorithms are derived from OpenEvals (MIT, © 2025 LangChain, Inc.) — see NOTICE.