Skip to content

Roster expansion Step 1: ingest 13 real supersession corpora (no gold labels) #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tcballard wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Roster expansion Step 1: ingest 13 real supersession corpora (no gold labels) #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 36 additions & 27 deletions decisiongrounding/docs/real-scenario-candidates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ distractor pool. This is the reportable complement to the synthetic bank in
`scenarios/` (which is auto-included in the sweep with no pre-registration
impact).

**This file is a candidate list only.** Nothing here is ingested into
`scenarios_real/` yet — a `scenario.json` under `scenarios_real/` would break
**Status: Step 1 done.** The confirmed corpora below are ingested into
`scenarios_candidates/` (corpus + provenance, no `scenario.json` yet) — see
that directory's `README.md`. They are inert until promoted: nothing under
`scenarios_candidates/` is loaded by the harness or counted by
`tests/test_real_roster.py::PINNED_ROSTER`. Accepting any candidate is a
**deliberate pre-registration event**: it lands together with a `PINNED_ROSTER`
update *and* a new analysis-plan amendment (see
Expand Down Expand Up @@ -44,31 +46,38 @@ The ingest tools write `corpus/*.md` + `provenance.json` only. The
satisfy the offline gate (`context_dump` adheres, `no_grounding` fails) and the
scoring mechanics documented in `spec/scoring-rubric.md`.

## Candidates
## Ingested candidates (Step 1 — confirmed edges)

Estimated band is a pre-ingest guess; confirm with the overlap helper after
ingest. Gold labels are intentionally left as **TODO — blind author**.
Each row is staged in `scenarios_candidates/<id>/` with a reproducible corpus +
provenance and **no gold label yet**. Every edge is derived from the artifacts'
own headers at the pin (`Replaces`/`Superseded-By` for PEPs, `Obsoletes` for
RFCs) — none are hand-asserted — and verified with
`python -m ingest.<tool> verify --out scenarios_candidates/<id>`. The overlap
band is an estimate from the topic; it is finalized when the blind task is
written and measured by `scenarios/overlap.py`.

| # | domain | ingest ids | proposed type | est. overlap | gold label |
|---|---|---|---|---|---|
| 1 | PEP typing | PEP 484 ← 483 | superseded_decision | high | TODO — blind author |
| 2 | PEP string formatting | PEP 3101 vs `%` formatting | prohibition_at_point_of_action | high | TODO — blind author |
| 3 | PEP packaging metadata | PEP 566 ← 345 | superseded_decision | high | TODO — blind author |
| 4 | PEP async | PEP 492 (async/await) vs generator coroutines | prohibition_at_point_of_action | high | TODO — blind author |
| 5 | PEP encodings | PEP 3120 (UTF-8 default source) | prohibition_at_point_of_action | medium | TODO — blind author |
| 6 | PEP division | PEP 238 (true division) supersedes classic | superseded_decision | medium | TODO — blind author |
| 7 | RFC HTTP | RFC 7231 ← 2616 (HTTP/1.1 semantics) | superseded_decision | medium | TODO — blind author |
| 8 | RFC TLS | RFC 8446 (TLS 1.3) prohibits legacy cipher suites | prohibition_at_point_of_action | medium | TODO — blind author |
| 9 | RFC email | RFC 5321 ← 2821 (SMTP) | superseded_decision | low | TODO — blind author |
| 10 | RFC URI | RFC 3986 ← 2396 (URI syntax) | superseded_decision | low | TODO — blind author |
| 11 | RFC auth | RFC 6749 (OAuth 2.0) prohibits the implicit grant in current guidance | prohibition_at_point_of_action | low | TODO — blind author |
| 12 | RFC DNS | RFC 8484 (DNS over HTTPS) vs cleartext DNS | conflicting_scoped | low | TODO — blind author |
| 13 | RFC datetime | RFC 3339 timestamp profile | prohibition_at_point_of_action | low | TODO — blind author |
| 14 | W3C | HTML5 supersedes an earlier dated edition | superseded_decision | low | TODO — blind author |
| 15 | W3C | CSS module leveling supersession | superseded_decision | low | TODO — blind author |
| candidate id | edge (source ← target) | proposed type | est. overlap |
|---|---|---|---|
| `peps_class_creation_supersession` | PEP-0487 ← 0422 | superseded_decision | high |
| `peps_typeis_narrowing_supersession` | PEP-0742 ← 0724 | superseded_decision | high |
| `peps_single_dispatch_supersession` | PEP-0443 ← 0245, 0246 | superseded_decision | high |
| `peps_metadata_v12_supersession` | PEP-0345 ← 0314 (← 0241) | superseded_decision | medium |
| `peps_package_layout_supersession` | PEP-0402 ← 0382 | superseded_decision | medium |
| `peps_hashlib_api_supersession` | PEP-0452 ← 0247 | superseded_decision | medium |
| `peps_packaging_governance_supersession` | PEP-0772 ← 0609 | superseded_decision | low |
| `peps_windows_installer_supersession` | PEP-0773 ← 0397, 0486 | superseded_decision | low |
| `peps_tls_api_supersession` | PEP-0748 ← 0543 | superseded_decision | low |
| `peps_pypi_mirror_supersession` | PEP-0449 ← 0381 | superseded_decision | low |
| `rfc_utf8_supersession` | RFC-3629 ← 2279 | superseded_decision | medium |
| `rfc_tcp_supersession` | RFC-9293 ← 0793 | superseded_decision | low |
| `rfc_ipv6_supersession` | RFC-8200 ← 2460 | superseded_decision | low |

Target distribution across the three discriminating types and three overlap
bands, so an accepted subset can fill the same matrix the synthetic bank
covers. Ingesting more candidates than are ultimately accepted is fine — only
those with a completed blind gold label and a passing offline gate join the
roster (and the roster pin).
Step 1 deliberately covers **supersession** edges only — they are mechanically
verifiable from headers. Prohibition and conflicting-scoped real scenarios
(which have no machine-readable edge) are a follow-on: the blind author selects
a document carrying a verbatim `MUST NOT`/prohibition and writes the task, using
the synthetic bank's prohibition/conflicting scenarios as the shape reference.

Ingesting more candidates than are ultimately accepted is fine — only those
with a completed blind gold label and a passing offline gate join the roster
(and the roster pin).
51 changes: 51 additions & 0 deletions decisiongrounding/scenarios_candidates/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Scenario candidates (staging area — NOT the reportable roster)

Real, pinned corpora ingested for the graded-overlap roster expansion, held
here **before** they join the frozen `scenarios_real/` roster. Each directory
has `corpus/*.md` + `provenance.json` (reproducible from the pin) but **no
`scenario.json`** — the task and gold label are authored later, blind
(CONTRIBUTING rule 1), ideally by an independent third party.

Nothing here is loaded by the harness or counted by the roster:
`scenarios/loader.py::load_scenarios` and `tests/test_real_roster.py` only see
directories containing a `scenario.json`. So these corpora are inert until
promoted.

## What's staged (Step 1 of the expansion)

13 real supersession edges, all verified to reproduce from their pins
(`python -m ingest.peps|rfcs verify --out scenarios_candidates/<id>`). The
overlap band is an *estimate* until the blind task is written and measured by
`scenarios/overlap.py`.

| candidate id | edge (source supersedes target) | est. overlap |
|---|---|---|
| `peps_class_creation_supersession` | PEP-0487 ← PEP-0422 | high |
| `peps_typeis_narrowing_supersession` | PEP-0742 ← PEP-0724 | high |
| `peps_single_dispatch_supersession` | PEP-0443 ← PEP-0245, PEP-0246 | high |
| `peps_metadata_v12_supersession` | PEP-0345 ← PEP-0314 (← PEP-0241) | medium |
| `peps_package_layout_supersession` | PEP-0402 ← PEP-0382 | medium |
| `peps_hashlib_api_supersession` | PEP-0452 ← PEP-0247 | medium |
| `peps_packaging_governance_supersession` | PEP-0772 ← PEP-0609 | low |
| `peps_windows_installer_supersession` | PEP-0773 ← PEP-0397, PEP-0486 | low |
| `peps_tls_api_supersession` | PEP-0748 ← PEP-0543 | low |
| `peps_pypi_mirror_supersession` | PEP-0449 ← PEP-0381 | low |
| `rfc_utf8_supersession` | RFC-3629 ← RFC-2279 | medium |
| `rfc_tcp_supersession` | RFC-9293 ← RFC-0793 | low |
| `rfc_ipv6_supersession` | RFC-8200 ← RFC-2460 | low |

All edges are derived from the artifacts' own `Replaces` / `Superseded-By`
(PEP) or `Obsoletes` (RFC) headers at the pin — none are hand-asserted.

## Promoting a candidate into the roster (Steps 2–3)

1. **Author `scenario.json` blind** (task + `gold_label`), per
`spec/external-scenario-request-graded-overlap.md`. Must pass the offline
gate (`context_dump` adheres, `no_grounding` fails) and match its declared
`lexical_overlap` band under `scenarios/overlap.py`.
2. `git mv scenarios_candidates/<id> scenarios_real/<id>` (provenance is
path-independent; re-run `verify` afterward).
3. Extend `tests/test_real_roster.py::PINNED_ROSTER` and its per-domain counts,
and land a new `spec/analysis-plan-amendment-*.md` re-registering the
enlarged roster — the deliberate pre-registration event. Amendment-1 and
amendment-2 stay frozen.
Loading
Loading