feat(config): support eval-level default expect checks

[roark47]

Background

Users want to define common expect checks once in eval.yaml instead of repeating the same block in every evals/cases/*.yaml file.

Today, expect is case-only:

• config.EvalConfig has no Expect field.
• config.CaseDefaults currently supports defaults such as timeout_seconds, max_turns, and collect_artifacts, but not expect.
• config.CaseConfig has Expect Expect yaml:"expect".
• defaultEvaluator.runExpectPreCheck() calls resolveExpectConfig(&caseCfg.Expect), so only per-case expect is evaluated.

User Need

Many eval suites have invariant zero-cost checks that should apply to every case, for example:

• exit_code: 0
• output must not contain generic failure text or placeholders;
• generated files must exist for every case;
• a common safety/format marker must be present or absent.

Repeating those checks in every case file is noisy and easy to drift. Since expect is the cheap pre-judge gate, making common checks reusable would reduce boilerplate while preserving the existing fast-fail behavior.

Possible Design

Prefer adding expect under cases.defaults, because it matches the existing case-defaults inheritance model:

# evals/eval.yaml
cases:
  files:
    - evals/cases/*.yaml
  defaults:
    timeout_seconds: 180
    max_turns: 1
    expect:
      exit_code: 0
      must_not_contain:
        - "TODO"
        - "I cannot"
      files_exist:
        - "result.json"

Each case could still add or override checks:

# evals/cases/special.yaml
id: special
input:
  prompt: Run the special task
expect:
  must_contain:
    - "SPECIAL_OK"

An alternative is a top-level expect: in eval.yaml, but cases.defaults.expect may be less ambiguous because expect is evaluated per case, after each case's agent run.

Merge Semantics To Define

The implementation should explicitly define how defaults and per-case expect combine. A reasonable starting point:

• If no case-level expect is set, use cases.defaults.expect.
• Slice fields such as must_contain, must_not_contain, files_exist, files_not_exist, and file_contains are appended/de-duplicated with defaults first.
• Scalar fields such as exit_code and golden_file can be overridden by the case when set.
• Existing case-only behavior remains backward compatible.
• expect still runs before judge, and a failing merged expect still skips judge execution.

Acceptance Criteria

• CaseDefaults or EvalConfig supports a documented eval-level way to define default expect checks.
• The evaluator applies the effective merged expect for every case.
• Per-case expect can add to or override defaults using documented rules.
• Existing evals with only case-level expect behave unchanged.
• Unit tests cover config loading, merge behavior, scalar override behavior, and expect pre-check execution.
• Documentation and templates are updated in docs/guide/writing-evals.md, docs/zh/guide/writing-evals.md, and skills/skill-upper references/templates.

[Keesan12]

Eval-level default checks are valuable because “done” should not depend on every task author remembering the same safety conditions.

We use the same idea in MartinLoop: a run can inherit a finish line, but the evidence still has to be visible in its receipt.

Would you let a task weaken inherited checks, or only add stricter ones? I would love your answer. We are building the open-source execution layer here: https://github.com/Keesan12/martin-loop