spec: refine — graded colleague work becomes training data with provenance [colleague#291/S6]

[OriNachum]

spec: refine — graded colleague work becomes training data with provenance (dataset pipeline v0)

Part of agentculture/colleague#291 holistic scoping (S6). Priority: Wave C. Depends on colleague S7 (the artifact/feedback export contract). Owner: data-refinery-cli (operator decision, agentculture/colleague#291, 2026-07-06).

Problem

The flywheel's middle stage does not exist anywhere: nothing turns colleague's graded work items into training data. data-refinery-cli is data-quality-in-storage (opaque Envelope store + validate/dedup/integrity/freshness) with zero transcript→dataset stages; daria's docs/pipeline.md describes the loop (Digest→Dream→Evaluate→Fine-Tune→Deploy) but is explicitly "Phase 2 — documented for future implementation" with no code; unsloth expects hand-authored JSONL. Meanwhile colleague already emits the raw material: run artifacts (request, steps, summary) + feedback records (1–5 grade).

Owner repo

data-refinery-cli (decided: its Envelope store + quality primitives are the natural substrate, and it already owns the pinnable-contract discipline — docs/contract.md v3). daria stays the autonomous-driver vision on top of these verbs, not the implementation.

Dependency repos

colleague (S7 export is the input contract), unsloth-cli (output must pass sloth validate / train --dry-run).

Proposed interface

New refine noun on data-refinery:

• data-refinery refine dataset --from <graded.jsonl> --schema chat --min-rating 4 --split 90/10 --out <dir> — consumes colleague's feedback export JSONL (one line per graded work item), maps each to a training example (request → assistant summary/diff in sloth's chat schema), filters on the grade threshold, splits train/eval disjoint by construction, writes train.jsonl + eval.jsonl.
• Every emitted example carries provenance metadata: {source: "colleague", task_id, rating, content_sha256, exported_at} — riding the Envelope metadata shape.
• refine lineage --dataset <dir> — reads back the provenance summary (which work items, grade distribution, dedup count).
• Internally reuses the existing quality primitives: dedup before split, integrity hashes on the emitted files, validate on the envelope shapes.

CLI examples

colleague feedback export --min-rating 4 --format jsonl > graded.jsonl   # S7
data-refinery refine dataset --from graded.jsonl --schema chat \
  --min-rating 4 --split 90/10 --out ds/colleague-v0
sloth validate --dataset ds/colleague-v0/train.jsonl                     # S4
data-refinery refine lineage --dataset ds/colleague-v0 --json

Acceptance criteria

• sloth train --dry-run accepts the produced train/eval JSONL verbatim.
• Every emitted example names its source work item (task_id, rating, hash) — an ungraded or below-threshold work item never silently enters a dataset (test).
• Train/eval splits are disjoint by construction (test: no shared content hash).
• The refine contract lands in docs/contract.md (v4 bump) with the example shape frozen.
• No LLM call, no network — the pipeline stays deterministic (the repo's existing discipline).

First implementation step

Failing test: a 3-line graded.jsonl fixture (ratings 5, 4, 2; threshold 4) → exactly 2 examples out, each carrying task_id+rating provenance, and the rating-2 item absent.

Non-goals

• No transcript synthesis (daria's Dream stage), no LLM-based cleaning, no reward modeling — v0 is a deterministic map/filter/split.
• No HF-datasets/alpaca/sharegpt writers (sloth's chat/task JSONL only).
• Does not decide when to train (S5) or what to serve (lobes).

Expected size

2–3 PRs (refine dataset; lineage + contract doc; quality-primitive integration).

• colleague (Claude)