Which LLM is the better family cook? An open-source LLM evaluation comparing GPT, Claude, Gemini, Kimi, DeepSeek, Qwen, Mistral and GLM on realistic weekly meal planning and recipe writing — with LLM-as-a-judge scoring, deterministic constraint checks, and a blind human taste rating.
A pydantic-evals benchmark that tests LLMs on a real, constraint-heavy task: planning a week of family dinners and writing the recipes — for a household with a weight-loss goal, a low-added-sugar requirement, a very picky 7-year-old, and a 1-year-old baby, under a 30-minute weekday cooking budget with a Sunday batch-prep session.
Generic benchmarks measure reasoning. This one measures whether a model can be an exciting cook while respecting hard constraints — the gap that makes most LLM meal plans boring.
For each contestant model in models.toml, the benchmark runs the same fixed set of test cases through this pipeline (built on pydantic-evals):
┌── contestant model (e.g. claude-sonnet-5) ──┐
test case ──────────► family brief + scenario + 4-week history │
(fixed inputs) │ must answer in a strict Pydantic schema │
└──────────────────┬──────────────────────────┘
│ WeeklyMenu / Recipe object
┌───────────────┴───────────────┐
▼ ▼
deterministic checks LLM judges (2 models,
(pure Python, free, 5 rubrics each, blind
objective pass/fail) to which chef answered)
└───────────────┬───────────────┘
▼
per-model score table (results/)
Every contestant sees the exact same inputs: the anonymized family brief (src/chef_bench/family.py) as its system prompt, plus the scenario text and dish history of each case. Output is forced into a Pydantic schema (src/chef_bench/models.py) — a model that can't produce a valid 7-dinner structure fails the case outright (schema validation, 2 retries). That alone filters out "sounds nice, isn't usable" answers.
The benchmark measures the two things a generic LLM meal plan usually gets wrong, separately:
- Constraint obedience — can the model respect hard family rules under pressure? Measured by deterministic checks: plain Python code inspecting the structured output. No AI involved, no judgment calls, fully reproducible.
- Cooking quality — is the plan actually good: exciting, varied, authentic, realistic? That can't be checked by code, so it's scored by LLM judges against written rubrics.
A model must win on both axes. A creative chef that ignores the 30-minute weekday budget is as useless as an obedient one that proposes plain pasta seven times.
v1 scored nutrition with a judge rubric — an LLM's opinion of an LLM. v2 makes nutrition measurable with a three-layer anti-cheat design. Models must declare per-portion macros (kcal, protein, carbs, sugars, fat, fiber), gram weights per ingredient, and food-group tags — then code verifies:
- Atwater consistency (free, no data needed): claimed kcal must match the kcal implied by the model's own claimed macros (4·protein + 4·carbs + 9·fat, ±15%). Fabricated numbers are usually internally incoherent.
- Plausibility bands: adult dinner portions must land in 350–900 kcal; scenario-specific budgets (sugar grams, kcal caps) are parsed from the case text and enforced per dinner.
- Ground-truth recomputation: recipe macros are recomputed from the
ingredient gram weights against a bundled food-composition table
(src/chef_bench/data/food_table.csv,
~140 ingredients, values approximated from USDA FoodData Central and ANSES
CIQUAL).
MacroAccuracy= 1 − mean relative error over kcal/protein/fat. If less than 70% of a recipe's grams can be matched, the check reportscoverage_too_lowand abstains — it never fakes a score.
Special-consideration checks run the same way in code: gluten/lactose ingredient scans for allergy cases, an iron-and-sodium guard for the baby cases, and an IBD flare-trigger scan (whole grains, legumes, nuts/seeds, raw-veg staples, personal triggers) paired with a ≥25 g protein floor — so a model cannot pass the restriction by simply serving less food.
Case-design methodology: every special-consideration case manipulates
exactly ONE condition on top of the constant family baseline, so failures are
attributable (factorial design). A single integration_week case stacks three
conditions at once and is reported separately — it measures constraint
integration as its own construct and is deliberately the hardest item.
Medical disclaimer: the IBD and other medical scenarios encode general published guidance (e.g. low-residue patterns during flares per ESPEN / Crohn's & Colitis foundation materials). Real medical nutrition is individualized; this is a benchmark for comparing language models, not medical or dietary advice.
Cases are organized into eval suites — each measures exactly one construct
so failures are attributable, and each can be run and re-run on its own
(chef-bench suites lists them; chef-bench run --suite <name> runs one).
Every case puts one manipulated condition on top of the constant family
baseline; two task formats exist (7-dinner weekly menus and full recipes with
gram-level ingredients).
The control condition every other suite is compared against.
standard_week: a normal week, no excuses — just a good, varied plan.recipe_weekday_turkish/recipe_fish/recipe_veggie: three everyday dishes as complete recipes — quantities, ≤30/20-min steps, baby adaptation.recipe_sunday_prep: ONE recipe for the whole 2-hour Sunday batch session — parallel-ordered steps, storage instructions, which dinner each component serves. Tests planning, not just cooking.
summer_week/winter_week: identical brief, opposite seasons. A model that serves the same stew in July as in January fails the construct.
variety_rescue: the 4-week history is pasta eight times and the family is explicitly bored — can the model escape a rut it's been anchored to?carte_blanche: rules loosened (any cuisine, longer weekdays), köfte/bulgur banned as the backbone. Measures range — v1 showed models converge on the brief's local optimum unless explicitly freed.
Did the model actually read the week?
absence_midweek: fewer portions Tue/Wed, a 15-minute dinner Thursday.guests_saturday: one impressive dinner without derailing the week.hectic_week: three ≤20-min days — Sunday prep must do real work.
low_sugar_stress: qualitative strict low-sugar week (judge-scored fit).diabetic_sugar_budget: hard target: ≤15 g sugars per adult portion, checked against the model's declared macros per dinner.calorie_target_week: hard target: ≤650 kcal per adult portion. All menus everywhere additionally pass Atwater consistency, calorie plausibility, and weekly food-group balance checks (see below).
allergy_gluten_free_week: a celiac guest for the whole week; dish names and components scanned for gluten carriers.recipe_gluten_free/recipe_lactose_free: full recipes whose ingredient lists are scanned against allergen keyword maps (with exceptions for e.g. rice noodles, coconut milk) — names carry no negations, so scanning is reliable where prose isn't.
picky_eater_stress: the 7-year-old refused everything new for two weeks — rebuild curiosity without pressure (child-feeding psychology, judge-scored, safe component still code-checked).recipe_baby_iron: the 1-year-old's portion is the priority — iron-rich component (keyword-verified), explicit salt handling, no honey/whole nuts.
ibd_flare_guest: a guest in an active Crohn's flare joins all 7 dinners. Low-residue rules (no raw veg, whole grains, legumes, nuts/seeds; personal triggers: onions, coffee, spice) apply to the guest's portion ONLY — the family still needs vegetables and fiber, so every dinner requires a statedspecial_adaptation. A ≥25 g protein floor blocks the lazy solution of simply serving less.recipe_ibd_flare: one dinner as a full recipe with the flare variant spelled out; ingredients scanned against the trigger map.
integration_week: IBD flare guest AND ≤15 g sugar AND ≤650 kcal — all at once. Passing the isolated suites but failing this one localizes the weakness precisely: constraint integration, not constraint knowledge. Reported as its own column, never averaged into the other suites.
Every non-smoke run upserts its per-case scores into results/store.json.
Rerunning one suite updates only those cases; everything else keeps its last
scores. chef-bench report renders the merged cross-suite table
(results/report.md) — models × suites, with - where a suite hasn't been
run for a model:
uv run chef-bench suites # what exists
uv run chef-bench run --suite medical,integration \
--contestant gpt-5.6-sol,kimi-k3 # rerun just these
uv run chef-bench report # merged state of the world- Each judge×rubric pair produces a 0–1 score per case; the two judges' scores for the same rubric are averaged, then averaged across cases.
- Deterministic checks average the same way (pass/fail → 1/0).
- The results table shows every metric per model plus a plain mean — so you can see why a model wins (e.g. "most creative but worst at the time budget"), not just that it wins.
- Raw per-case data lands in
results/results.json; the table inresults/summary.md.
Every run also writes results/human-rating.md: all menus and recipes with
the chefs anonymized ("Chef A", "Chef B", ...) with stable letters across
cases, so a chef can be compared between scenarios without knowing the model. Rate each answer's creativity 1–5, then
open results/human-rating-key.json to de-anonymize. The family's own taste
is the tie-breaker the LLM judges can't provide.
Every output is scored by two judges from different labs (configured in models.toml) and averaged; judges see only the case input and the structured output — never which model produced it. Deterministic checks are unaffected by judge choice entirely. This is a mitigation, not a solution: a judge may still favor its own lab's writing style, which is why half the final signal comes from code, not judges.
cp .env.example .env # add your OpenRouter API key
uv sync
uv run chef-bench validate # check the model ids in models.toml are live
uv run chef-bench run --smoke # 1 case, 1 model — cheap end-to-end check
uv run chef-bench run # the full benchmarkContestants and judges are configured in models.toml — edit that
file, nothing else, to change the lineup. Results land in results/
(results.json + summary.md).
Offline tests (no API calls): uv run pytest
Human blind pre-selection (10 models, creativity rated by the family) picked 3 finalists (gpt-5.6-sol, kimi-k3, claude-sonnet-5); the full judged benchmark then ranked them across all 10 menu scenarios and 4 recipes:
| model | HasSundayPrepReuse | NoHistoryRepeats | NoPorkMains | SevenDistinctDays | SonSafeEveryDay | WeekdayTimeBudget | interest | nutrition_fit | practicality | turkish_med | variety | mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| claude-sonnet-5 | 0.90 | 1.00 | 1.00 | 1.00 | 0.99 | 1.00 | 0.91 | 0.97 | 0.91 | 0.92 | 0.83 | 0.949 |
| gpt-5.6-sol | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.98 | 0.94 | 0.97 | 0.94 | 0.88 | 0.90 | 0.965 |
| kimi-k3 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.98 | 0.95 | 0.98 | 0.93 | 0.88 | 0.88 | 0.964 |
| model | HasBabyAdaptation | IngredientsHaveQuantities | ReasonableStepCount | WithinStatedTimeBudget | recipe_clarity | recipe_quality | mean |
|---|---|---|---|---|---|---|---|
| claude-sonnet-5 | 1.00 | 0.90 | 1.00 | 1.00 | 0.84 | 0.91 | 0.941 |
| gpt-5.6-sol | 1.00 | 0.96 | 1.00 | 1.00 | 0.89 | 0.95 | 0.967 |
| kimi-k3 | 1.00 | 0.92 | 1.00 | 1.00 | 0.93 | 0.94 | 0.965 |
Winner: gpt-5.6-sol — top of both suites and the family's blind-rating favorite (5/5). kimi-k3 finished in a statistical tie at a fraction of the price; claude-sonnet-5 third (only finalist to miss a Sunday-prep reuse, and lowest on variety and recipe clarity). All three were essentially perfect on the hard constraints, so the ranking is decided by cooking quality — which is the point of the benchmark.
Notable: in the human pre-selection, the most expensive contestant (gemini-3.1-pro) did not make the family's cut, and the budget wildcard kimi-k3 tied the frontier models. Taste is not priced in dollars.
Full data: results/finals.json; blind pre-selection sheets and keys in
results/.
The family brief in src/chef_bench/family.py is deliberately generalized — no names, no locations, no identifying details.
MIT