Distributed multi-seed / multi-eval: served path runs serially on ONE endpoint; no seed-sharding fan-out

[penfever]

Summary

Our served-endpoint eval path (Marin's serve_and_eval.py + the evalchemy local-completions/local-chat-completions client) runs a whole multi-seed / multi-eval job serially against ONE served vLLM endpoint. For slow legs this is many hours end-to-end:

• AIME24 10-seed on a thinking/large model ≈ 50–80 min/seed → 8–13 h wall-clock for the 10 sequential passes.
• Big tier-2 suites (MATH500 / HumanEvalPlus / MBPPPlus / GPQADiamond / IFEval) on a 67B likewise run task-after-task against a single endpoint.

We want a proper distributed path: split seeds / tasks / repetitions across multiple serve endpoints (data-parallel across GPUs/nodes) so each shard finishes fast and in parallel, then aggregate — instead of one long serial run guarded by an ever-growing wall-clock backstop.

Today's concrete failure (what motivated this)

The served backend self-stops after a wall-clock lifetime (SERVE_TIMEOUT_HOURS) as a backstop against an orphaned server. That backstop was 4.0 h, which is shorter than a single slow multi-seed run, so it killed LIVE endpoints mid-eval. The evalchemy client then scored the remaining requests as connection-refused MISSES → degenerate 0.0-tail results on three legs today: 35B AIME, 80B AIME, 67B tier-2.

Band-aid already applied (Marin wrapper, branch feuer/marinbase-eval-cw-driver): SERVE_TIMEOUT_HOURS bumped 4.0 → 24.0 h (experiments/evals/evalchemy/serve_and_eval.py:89). Marin main still carries the 4.0 h default. This only widens the timeout window; it does not shorten wall-clock. The real fix is to distribute the work.

What evalchemy supports for splitting TODAY (with proof)

1. Request concurrency to ONE endpoint — num_concurrent in --model_args (parallel in-flight HTTP requests).

• Default 16: serve_and_eval.py:91 (DEFAULT_NUM_CONCURRENT); threaded into model_args at run_evalchemy_client.py:47, and eval/serve_eval/run.py builds the same for its endpoint provider.
• Ceiling = a single endpoint's throughput. Does not scale past one server.

2. Per-task processes (in our wrapper), but still sequential — the Marin client runs one eval.eval subprocess per task, in a plain Python for loop, against the single endpoint:

• experiments/evals/evalchemy/run_evalchemy_client.py:217 — for task in tasks: … subprocess.run(cmd).
• Within a single eval.eval, generation across tasks is itself sequential: eval/eval.py:235 — "Run benchmark evaluations - sequential generation, parallel evaluation" (only the metric step is parallel).

3. Native per-problem repetition (pass@k) — --num_samples (>1) + --pass_at_k, generating N completions per problem, aggregated via estimate_pass_at_k.

• eval/eval.py:150-161 (args), eval/passk.py (aggregator). Still runs within one process against one endpoint; only num_concurrent parallelizes the extra requests.

4. Data-parallel DATASET-SHARD path — eval/distributed/ is the one true multi-worker fan-out, but it is a separate launch path disjoint from the served path:

• eval/distributed/launch.py — --num_shards (default 128) submits a SLURM array job (--array=0-{num_shards-1}, launch.py:346), one rank per shard.
• eval/distributed/process_shard.py:83-87 — shuffles then ds.shard(num_shards=global_size, index=rank): shards by dataset row.
• process_shard.py:92-97 — each rank spins up its own in-process vllm.LLM(...) (NOT a served OpenAI endpoint client).
• Aggregation = upload parquet shards to HF → rescore with --model precomputed_hf (launch.py:639).
• It shards ROWS, not seeds. Per-example seed is only read from gen_kwargs.get("seed") at process_shard.py:115, but launch.py's dataset creation does no seed/repetition expansion — nothing multiplies the dataset by N seeds or assigns per-row seeds.

5. Tensor parallel (--tp / TP=8) — model-parallel across GPUs of ONE serve; not a throughput scale-out of independent shards.

What evalchemy does NOT support — the seed dimension (operator's "can't split by seed" — CONFIRMED)

--seed sets a single global 4-tuple (python/numpy/torch/fewshot) for one eval.eval process — it is a per-run override, not a repetition multiplier:

• eval/lm_eval_compat.py:247-260 — --seed = 4-value tuple or single int, one global override.

Consequently a multi-seed AIME μ±σ run is expressed as N independent seeded passes, and in the Marin wrapper they are N EvalTaskConfig entries run sequentially against the one endpoint:

• experiments/evals/evalchemy/marin_evalchemy_gpu.py:197-201 — EVAL_TASK_SET=aime24_seeds builds AIME24 × seeds 42..51 (10 entries, distinct task_alias/task_kwargs["seed"]).
• Per-task --seed passthrough: serve_and_eval.py:402-403, run_evalchemy_client.py:99-100.
• These 10 entries are consumed by the serial for task in tasks loop (evidence [OlympiadBench] Add separate official English text-only task #2) → 10 sequential passes on ONE endpoint.

There is no seed-sharding / seed-fan-out anywhere — not in the served path (no fan-out at all) and not in the eval/distributed/ path (which shards rows, and whose launcher does not even expand seeds into rows). So: evalchemy can split by dataset-shard, by task, by request-concurrency, and by pass@k repetition; it cannot split by seed, and the served-endpoint path has no multi-worker fan-out of any kind.

The served path has no fan-out (root cause)

Marin's production path is one parent → one serve child + one eval child:

• serve_and_eval.py:438-439 — with serve_model(...) as endpoint: _submit_eval_child(...) — a single endpoint, a single eval client.
• serve_and_eval.py:82-89 — the SERVE_TIMEOUT_HOURS comment itself documents that this serial design is what makes a slow run outlast the backstop.

Proposed design (scope only — do not implement here)

Goal: N serve endpoints, shard the seed/task/repetition workload across them, aggregate.

Option A — served-endpoint fan-out (fits our Marin production path).

1. evalchemy: add a served-path work-splitter that partitions a (task × seed × sample) job list into K shards and runs each shard's eval.eval against a different base_url, then merges the per-shard results_*.json (mean±σ over seeds; pass@k over samples). Natural home: a new eval/serve_eval/ distributed driver, or a --num_endpoints/--base_urls fan-out in the client. Seed becomes a first-class shard key (each (task, seed) an independent unit), which is exactly the dimension missing today.
2. Marin wrapper: serve_and_eval grows from one serve child to K serve children (K endpoints), and the eval child (or K eval children) dispatches shard i to endpoint i. Aggregation stays where EvalchemyResult reads the per-task_alias dirs. Each shard finishes in ~1/K the wall-clock, so the SERVE_TIMEOUT_HOURS backstop stops being load-bearing.

Option B — reuse eval/distributed/ for the served/seed case. Teach eval/distributed/launch.py to expand seeds/repetitions into dataset rows (assign per-row gen_kwargs.seed, replicate each problem × N seeds) so the existing row-sharder distributes seed work for free, then aggregate μ±σ per seed-group at rescore. This reuses the proven merge path but keeps the vLLM-per-shard model (not our served/MoE-fork path), so it's the better fit for HPC-array clusters than for our CoreWeave served path.

Recommendation: Option A for the Marin served path (MoE/exotic-arch models need the served vLLM-fork backend), optionally Option B's seed-expansion as the aggregation primitive.

Acceptance

• A multi-seed AIME (10 seeds) or a tier-2 suite runs across K endpoints in ~1/K wall-clock, with correct μ±σ / pass@k aggregation, and without relying on a large SERVE_TIMEOUT_HOURS to survive.
• Seed is a shardable unit.

References (evalchemy fork, this repo)

• eval/eval.py:150-161 (--num_samples/--pass_at_k), :235 (sequential generation)
• eval/lm_eval_compat.py:247-260 (--seed single global 4-tuple)
• eval/passk.py (pass@k aggregator)
• eval/distributed/launch.py (--num_shards, sbatch array :346, rescore :639), eval/distributed/process_shard.py:83-97,115 (row-shard + own vLLM + per-row seed)
• eval/serve_eval/run.py (served-endpoint driver, num_concurrent)

References (Marin wrapper, repo marin-community/marin, branch feuer/marinbase-eval-cw-driver)

• experiments/evals/evalchemy/serve_and_eval.py:82-91 (SERVE_TIMEOUT_HOURS 24.0 band-aid + rationale), :402-403 (seed passthrough), :438-439 (one serve + one eval)
• experiments/evals/evalchemy/run_evalchemy_client.py:217 (serial task loop), :47,99-100 (num_concurrent, --seed)
• experiments/evals/evalchemy/marin_evalchemy_gpu.py:197-201 (AIME24 seeds 42–51)
• NOTE: Marin main still ships SERVE_TIMEOUT_HOURS = 4.0; the 24.0 bump lives only on feuer/marinbase-eval-cw-driver.

[rjpower]

@penfever is the slowdown on the inference side? We have the broker functionality to load-balance across vLLM endpoints -- that's relatively easy to bring up instead of splitting here

it sounds like the seed issue might be a real separate thing

[penfever]

@rjpower I haven't done careful per-model benchmarking, but overall the throughput numbers for vLLM look to be in the ranges I would expect on this hardware.

The split-on-seed issue is indeed an Evalchemy limitation, I believe.

I guess the meta-point I want to raise here, and we discussed this awhile back, is that Evalchemy is not structured to take full advantage of the producer / consumer queue design patterns modern inference workflows use and this makes it slower than it has to be, which one tends to feel more on benchmarks like AIME.