Skip to content

Calibrate optimization candidate scoring with raw-free funnel evidence #2

Description

@alexandrefimov

Optimization Candidate scoring should be calibrated from structured analyzer facts and raw-free optimizer funnel evidence, not by loosening prompts, validators, or browser wording to increase draft count.

Scope:

  • measure the optimizer funnel on fixture and sanitized real corpora with raw-free counters;
  • separate expensive query, stats/query-context signal, recipe detected, safe to attempt, trusted draft, no-rewrite, recommendations-only, and no-draft reason;
  • use structured analyzer facts where available, with rendered markdown only as compatibility fallback;
  • keep no-recipe review tracks as telemetry and backlog routing, not recipe detection or proof of benefit;
  • preserve strict SQL validation and the no-echo browser trust boundary.

Acceptance criteria:

  • A raw-free calibration output records candidate, recipe-detected, draft-ready, trusted-draft, no-rewrite, recommendations-only, and failure/no-draft reason counts.
  • Any scoring or cap change is backed by focused fixtures and regression tests.
  • Browser/trusted-output tests prove no raw SQL, identifiers, predicates, local paths, artifact filenames, model names, or subprocess output leaks through optimizer surfaces.
  • The calibration distinguishes recipe coverage limits from model/prompt failures.
  • Documentation/changelog drift is checked and updated if scoring semantics or visible optimizer states change.

Maintainer notes:

This is maintainer-owned because it can change product ranking, optimizer trust semantics, and safety-sensitive visible states. Do not open a PR without agreeing on the calibration dataset shape and raw-free output contract first.

Metadata

Metadata

Assignees

No one assigned

    Labels

    analyzerDeterministic facts, scoring, or evidence extraction.enhancementBounded feature request or workflow improvement.maintainer-ownedNeeds maintainer discussion before PR; product, architecture, or trust-boundary owned.optimizerQuery Optimizer or details-page LLM optimizer behavior.safetyPrivacy, redaction, validation, or trust-boundary concern.testsTest coverage, fixtures, CI, or release-gate checks.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions