Skip to content

feat(benchmarks): reproducible runner for four benchmarks, with the multi-round decider made visible - #425

Open
YueAWu wants to merge 8 commits into
mainfrom
feat/llm-multiround-retrieval
Open

feat(benchmarks): reproducible runner for four benchmarks, with the multi-round decider made visible#425
YueAWu wants to merge 8 commits into
mainfrom
feat/llm-multiround-retrieval

Conversation

@YueAWu

@YueAWu YueAWu commented Aug 28, 2026

Copy link
Copy Markdown

Summary

One runner for four long-term-memory benchmarks (LoCoMo, LongMemEval,
EverMemBench, SubtleMemory), and the multi-round decider's silent failure paths
made visible.

The decider previously fell back to a fixed top-ranked core on any failure and
returned HTTP 200 with a full episode list, so a run whose decider never
answered reported a plausible number and exited zero. It now logs at error,
surfaces on SearchData.degraded, and the harness sends one real completion to
the decider before the first question and refuses to start if it does not
answer. Twelve sweep arms were measured against that degraded path before this.

Also: each config now names the arm its published number came from, [answer]
and [judge] models included; --smoke no longer discards an explicit --conv;
the EverMemBench converter moved into its adapter so the release's
session-numbering rule is written down once instead of twice.

Area

  • Architecture method
  • Benchmark
  • Use case
  • Documentation
  • Developer experience
  • CI, build, or release

Verification

make lint                       clean
pytest tests/unit               2815 passed, 8 deselected
benchmarks/audit/check_protocol.py   graded protocol matches each harness

End to end, following benchmarks/README.md with steps 1-3 already done:
  DATASET=locomo CONV=0 bash benchmarks/reproduce.sh --smoke
  -> 100.0% (10/10), all four stages, 1 conversation

Config resolution under `env -i` for all four benchmarks; both newly named
answer models return content on a real completion.

Converting the EverMemBench release after moving the converter reproduces the
shipped evermembench.json byte for byte.

juwei.yue added 8 commits August 27, 2026 09:53
…isible

Twelve retrieval arms were measured against a decider that answered 404 on
every call. The loop caught the error, fell back to a fixed top-3 core with a
single round, and returned HTTP 200 with a full episode list -- so the runs
reported plausible accuracies and exited zero. Nothing in the result said the
decider had never run.

Two causes, both fixed here:

Twelve `EVEROS_LLMMR_*` module constants were read at import time, so the
harness env that was supposed to configure them arrived too late and every
run used the defaults. They are now fields on `DeciderSettings`, resolved per
search through `_tuning()`, with the legacy env names still honoured.

The fallback logged at debug and was indistinguishable from a normal result.
It now logs at error with the underlying exception and calls `mark_degraded`,
which surfaces on `SearchData.degraded` -- a caller can tell a degraded result
from a healthy one without reading a trace file.

Degradations reset per search and restore around it, so a long-lived process
does not accumulate another request's reasons.
A 400 or 404 from the embedding endpoint was classified the same as a 503 and
went through the full retry ladder, turning an unservable model name into
minutes of backoff per call and, under concurrency, into an apparent hang.

`_classify` now maps 400/401/403/404/413/414/422 to `EmbeddingInputError` --
a domain error the caller can act on -- and leaves 408/429/5xx and transport
failures as `EmbeddingServiceError`, which is what retrying is for.
Three defects that only appear under a long benchmark run.

An OME strategy with no timeout could hold its slot indefinitely; a stalled
extraction then starved every later run behind it. Runs now carry
`run_timeout` and raise `TimeoutError`, which the dispatcher records, rather
than being cancelled with no trace.

The SQLite engine leaked a connection per thread -- the signature was the pool
growing from 6 to 58 and the connector wedging. The pool is now explicit and
sized, so exhaustion fails loudly instead of degrading.

The LLM client had no way to pass provider-specific fields, so the one setting
a thinking model needs to answer within a timeout could not be configured at
all. `extra` is now plumbed through.
Profile extraction listened on two triggers, one of which fired from the
clustering path, so a store built without clusters silently produced no
profiles and a store with them produced duplicates. It now listens on
`EpisodeExtracted` alone.

The clustering path itself is untouched -- `agentic` retrieval needs it, and
removing it is what left four rebuilt stores with zero clusters and an
`agentic` route that early-returned an empty set without erroring.

The profile lock is now per subject rather than global, so two subjects no
longer serialise behind each other.
The four benchmarks each had their own driver, so a fix to one never reached
the others and no two numbers were produced by the same code. They now share
`run.py` and differ only in `adapters/<name>.py`, with every knob that decides
a number living in `configs/<dataset>.toml` -- a run is reproducible from its
config, and `reproduce.sh` passes no model overrides.

Defects this closes, each of which produced a complete-looking wrong number:

Profile injection was implemented only in the EverMemBench adapter; the other
three accepted `include_profile` and discarded the profiles. The rendering is
now shared, and returns the memories unchanged when there is no profile so
existing prompts stay byte-identical.

A decider model name with no endpoint went to the extraction endpoint, 404ed
on every call, and fell back silently. `run.py` now probes the decider with a
real call before the first question and refuses to start if it does not
answer; `--decider-base-url` exists so the pair can be set together, and both
are folded into `run_spec.json` so it records what actually ran.

A store whose owners were built under different partition keys returned zero
episodes and scored ~0.5% with no error. SEARCH now asserts the owner exists
in the store first, and names what the store does hold when it does not.

`.env.example` shipped path placeholders that looked configured, which beat
the config defaults and sent runs at `/01/dialogue.json`. Unresolved `${VAR}`
is now detected and reported by variable name.
…point

The README said the decider "runs the same model as extraction, which needs no
extra configuration". The configs disagree: `decider_model` defaults to the
published `qwen3.6-27B`, and `decider_base_url` deliberately has no default,
because no endpoint we could name would serve that model for the reader.

So a first run prints a warning the README never mentions -- that the decider has
no endpoint, that it is falling back to the `[llm]` model, and that the result is
therefore not comparable to the published numbers. Both READMEs now show that
banner and say plainly that such a run is valid but is not the published arm.

They also described only half the startup check. `run.py` has two paths: an
unresolved endpoint degrades with the warning, while a model pointed at an
endpoint that does not serve it aborts. Documenting only the abort left the
degrade looking like a bug.

The English README also carried its Chinese pointer line in Chinese, which
`check-cjk` counts against the English-first policy; the file's own mirror is
the allowlisted place for that.
…adapter

The README had grown into prose about why the harness is built the way it is:
sections on reproducibility, on what happens when a path is wrong, on managing
servers by hand. None of it told a reader what to type. It is now five numbered
steps -- install, data, environment, run, output -- plus a flag table, and 110
lines instead of 284. The per-role key table replaces the decider section: the
runner already prints what it fell back to and aborts on the endpoint mistake,
so the guide does not need to narrate either.

`convert_evermembench.py` moves into `adapters/evermembench.py`, which is where
it belonged: the adapter already reads the raw release to translate the gold
session names, so the release's session-numbering rule was written down twice --
the converter counting forward and rolling back on an all-blank pair, the
adapter skipping it up front. Equivalent on the released data (3,570 sessions
either way, verified), but that rule decides gold alignment, and if the two ever
drift the ids shift by one, gold matches the wrong session, and every stage still
reports success. Both now walk `iter_raw_sessions`. Converting the release again
after the move reproduces the shipped `evermembench.json` byte for byte.

`--smoke` also stops discarding an explicit `--conv`. It assigned `[0, 1]`
unconditionally, so `--conv 0 --smoke` ran two conversations and scored 20
questions; the banner printed the conversations it had chosen rather than the
ones it was given, so the only way to catch it was to count graded rows.
…all four

The four configs named the standard arm, not the published best one: LoCoMo ran
the 27B decider rather than deepseek-v4-flash-0731, and LongMemEval and
EverMemBench answered with gpt-4.1-mini rather than gemini-3.6-flash and
gemini-3-flash-preview. Following the README reproduced a number several points
below the one it cited. Each config now names the arm its published number came
from, and README section 6 states that number next to the decider and answer
model that produced it.

SubtleMemory needed no change: `answer_route` in its adapter already routes each
question between the two answer contracts, so the config was the published arm
already.

`check_protocol.py` pinned LongMemEval's answer model and nothing else, so it
reported "no divergence" for three configs it never looked at -- the two answer
models above changed under it without a word. It now covers all four, and the
values are the expanded ones the loader produces rather than the `${VAR:-...}`
templates the files carry, because that is what a run actually uses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant