feat(benchmarks): reproducible runner for four benchmarks, with the multi-round decider made visible - #425
Open
YueAWu wants to merge 8 commits into
Open
feat(benchmarks): reproducible runner for four benchmarks, with the multi-round decider made visible#425YueAWu wants to merge 8 commits into
YueAWu wants to merge 8 commits into
Conversation
added 8 commits
August 27, 2026 09:53
…isible Twelve retrieval arms were measured against a decider that answered 404 on every call. The loop caught the error, fell back to a fixed top-3 core with a single round, and returned HTTP 200 with a full episode list -- so the runs reported plausible accuracies and exited zero. Nothing in the result said the decider had never run. Two causes, both fixed here: Twelve `EVEROS_LLMMR_*` module constants were read at import time, so the harness env that was supposed to configure them arrived too late and every run used the defaults. They are now fields on `DeciderSettings`, resolved per search through `_tuning()`, with the legacy env names still honoured. The fallback logged at debug and was indistinguishable from a normal result. It now logs at error with the underlying exception and calls `mark_degraded`, which surfaces on `SearchData.degraded` -- a caller can tell a degraded result from a healthy one without reading a trace file. Degradations reset per search and restore around it, so a long-lived process does not accumulate another request's reasons.
A 400 or 404 from the embedding endpoint was classified the same as a 503 and went through the full retry ladder, turning an unservable model name into minutes of backoff per call and, under concurrency, into an apparent hang. `_classify` now maps 400/401/403/404/413/414/422 to `EmbeddingInputError` -- a domain error the caller can act on -- and leaves 408/429/5xx and transport failures as `EmbeddingServiceError`, which is what retrying is for.
Three defects that only appear under a long benchmark run. An OME strategy with no timeout could hold its slot indefinitely; a stalled extraction then starved every later run behind it. Runs now carry `run_timeout` and raise `TimeoutError`, which the dispatcher records, rather than being cancelled with no trace. The SQLite engine leaked a connection per thread -- the signature was the pool growing from 6 to 58 and the connector wedging. The pool is now explicit and sized, so exhaustion fails loudly instead of degrading. The LLM client had no way to pass provider-specific fields, so the one setting a thinking model needs to answer within a timeout could not be configured at all. `extra` is now plumbed through.
Profile extraction listened on two triggers, one of which fired from the clustering path, so a store built without clusters silently produced no profiles and a store with them produced duplicates. It now listens on `EpisodeExtracted` alone. The clustering path itself is untouched -- `agentic` retrieval needs it, and removing it is what left four rebuilt stores with zero clusters and an `agentic` route that early-returned an empty set without erroring. The profile lock is now per subject rather than global, so two subjects no longer serialise behind each other.
The four benchmarks each had their own driver, so a fix to one never reached
the others and no two numbers were produced by the same code. They now share
`run.py` and differ only in `adapters/<name>.py`, with every knob that decides
a number living in `configs/<dataset>.toml` -- a run is reproducible from its
config, and `reproduce.sh` passes no model overrides.
Defects this closes, each of which produced a complete-looking wrong number:
Profile injection was implemented only in the EverMemBench adapter; the other
three accepted `include_profile` and discarded the profiles. The rendering is
now shared, and returns the memories unchanged when there is no profile so
existing prompts stay byte-identical.
A decider model name with no endpoint went to the extraction endpoint, 404ed
on every call, and fell back silently. `run.py` now probes the decider with a
real call before the first question and refuses to start if it does not
answer; `--decider-base-url` exists so the pair can be set together, and both
are folded into `run_spec.json` so it records what actually ran.
A store whose owners were built under different partition keys returned zero
episodes and scored ~0.5% with no error. SEARCH now asserts the owner exists
in the store first, and names what the store does hold when it does not.
`.env.example` shipped path placeholders that looked configured, which beat
the config defaults and sent runs at `/01/dialogue.json`. Unresolved `${VAR}`
is now detected and reported by variable name.
…point The README said the decider "runs the same model as extraction, which needs no extra configuration". The configs disagree: `decider_model` defaults to the published `qwen3.6-27B`, and `decider_base_url` deliberately has no default, because no endpoint we could name would serve that model for the reader. So a first run prints a warning the README never mentions -- that the decider has no endpoint, that it is falling back to the `[llm]` model, and that the result is therefore not comparable to the published numbers. Both READMEs now show that banner and say plainly that such a run is valid but is not the published arm. They also described only half the startup check. `run.py` has two paths: an unresolved endpoint degrades with the warning, while a model pointed at an endpoint that does not serve it aborts. Documenting only the abort left the degrade looking like a bug. The English README also carried its Chinese pointer line in Chinese, which `check-cjk` counts against the English-first policy; the file's own mirror is the allowlisted place for that.
…adapter The README had grown into prose about why the harness is built the way it is: sections on reproducibility, on what happens when a path is wrong, on managing servers by hand. None of it told a reader what to type. It is now five numbered steps -- install, data, environment, run, output -- plus a flag table, and 110 lines instead of 284. The per-role key table replaces the decider section: the runner already prints what it fell back to and aborts on the endpoint mistake, so the guide does not need to narrate either. `convert_evermembench.py` moves into `adapters/evermembench.py`, which is where it belonged: the adapter already reads the raw release to translate the gold session names, so the release's session-numbering rule was written down twice -- the converter counting forward and rolling back on an all-blank pair, the adapter skipping it up front. Equivalent on the released data (3,570 sessions either way, verified), but that rule decides gold alignment, and if the two ever drift the ids shift by one, gold matches the wrong session, and every stage still reports success. Both now walk `iter_raw_sessions`. Converting the release again after the move reproduces the shipped `evermembench.json` byte for byte. `--smoke` also stops discarding an explicit `--conv`. It assigned `[0, 1]` unconditionally, so `--conv 0 --smoke` ran two conversations and scored 20 questions; the banner printed the conversations it had chosen rather than the ones it was given, so the only way to catch it was to count graded rows.
…all four
The four configs named the standard arm, not the published best one: LoCoMo ran
the 27B decider rather than deepseek-v4-flash-0731, and LongMemEval and
EverMemBench answered with gpt-4.1-mini rather than gemini-3.6-flash and
gemini-3-flash-preview. Following the README reproduced a number several points
below the one it cited. Each config now names the arm its published number came
from, and README section 6 states that number next to the decider and answer
model that produced it.
SubtleMemory needed no change: `answer_route` in its adapter already routes each
question between the two answer contracts, so the config was the published arm
already.
`check_protocol.py` pinned LongMemEval's answer model and nothing else, so it
reported "no divergence" for three configs it never looked at -- the two answer
models above changed under it without a word. It now covers all four, and the
values are the expanded ones the loader produces rather than the `${VAR:-...}`
templates the files carry, because that is what a run actually uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One runner for four long-term-memory benchmarks (LoCoMo, LongMemEval,
EverMemBench, SubtleMemory), and the multi-round decider's silent failure paths
made visible.
The decider previously fell back to a fixed top-ranked core on any failure and
returned HTTP 200 with a full episode list, so a run whose decider never
answered reported a plausible number and exited zero. It now logs at error,
surfaces on
SearchData.degraded, and the harness sends one real completion tothe decider before the first question and refuses to start if it does not
answer. Twelve sweep arms were measured against that degraded path before this.
Also: each config now names the arm its published number came from,
[answer]and
[judge]models included;--smokeno longer discards an explicit--conv;the EverMemBench converter moved into its adapter so the release's
session-numbering rule is written down once instead of twice.
Area
Verification