Skip to content

eval: schema advertises language as optional (no description) but parseEvalRequest rejects every omission #1395

Description

@NEWBIE0413

What happened?

In a live session the eval tool rejected every call with eval run requires language, leaving the agent with no execution surface (commits, tests, builds all blocked). Transcript forensics: 15 of 58 eval calls failed; every fully-captured failing call sent {code, summary} with no language, while the successful ones included it. The model kept omitting the field because the advertised schema says that's fine:

packages/senpi-codemode/src/tool/types.ts (createEvalInputSchema):

language: Type.Optional(languageSchema),   // no description

packages/senpi-codemode/src/tool/eval-request.ts (parseEvalRequest):

if (!isEvalLanguage(params.language)) throw new TypeError("eval run requires language");

So an omitted language is schema-valid but dies in execute(). The union with peek/stop presumably can't mark it cleanly required — but unlike summary (same required-but-schema-optional situation, resolved via its "REQUIRED for run..." description), language carries no signal at all. Two aggravators:

  • The same TypeError fires for a missing and an invalid language, so an agent debugging its own call can't tell which happened (mine concluded the harness was dropping its parameters).
  • prepareArguments in eval-tool.ts already repairs/clamps summary but does nothing for language.

Steps to reproduce

import { parseEvalRequest } from "…/senpi-codemode/src/tool/eval-request.ts";
parseEvalRequest({ code: "1", summary: "s" }); // throws, though the schema validates the same params

Or run any codemode session and watch a model that omits language — every cell fails identically.

Expected behavior

Maintainer's call, but in increasing size:

  1. Minimal, follows the summary convention: a description on languageSchema ("REQUIRED for run. ...").
  2. Split the error so missing vs invalid are distinguishable.
  3. Design question rather than assumption: should prepareArguments default an omitted language to the first enabled one? It would have kept the session alive, but changes semantics (py code with an omitted language would run in the js kernel).

Version

senpi-codemode 2026.9.5 (npm); verified still present on main today.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions