Skip to content

Integrated walkthrough, regression matrix, and CI gate - #6

Merged
maximalfocus merged 1 commit into
mainfrom
issue/5-walkthrough-and-release
Aug 16, 2026
Merged

Integrated walkthrough, regression matrix, and CI gate#6
maximalfocus merged 1 commit into
mainfrom
issue/5-walkthrough-and-release

Conversation

@maximalfocus

Copy link
Copy Markdown
Owner

Turns the two verified services into a finished local teaching artifact: one deterministic
walkthrough, an explicit regression matrix, educational documentation, and a CI job that runs the
same gate the README documents.

The walkthrough

ALLOW_VULNERABLE_DEMO=true docker compose --profile vulnerable run --rm walkthrough

Real run against the containers:

fieldblind walkthrough — FICTIONAL LOCAL DEMO — mode: full
claim EXP-204, every value below is fictional demonstration data

[PASS] vulnerable read disclosure
    actor            : niko (employee, owner)
    object verdict   : allowed — niko owns EXP-204
    property verdict : none applied — the whole object was serialized
    http outcome     : 200
    response keys    : claim_id, ..., risk_score, reviewer_note, decision, approved_amount_cents
    state diff       : (unchanged)

[PASS] secure read projection            ... response keys: the seven employee properties only
[PASS] vulnerable mass assignment        ... state diff: purpose, decision, approved_amount_cents
[PASS] secure whole-request rejection    ... http outcome: 400 generic, 1 audit event; state unchanged
[PASS] secure legitimate employee edit   ... state diff: purpose
[PASS] secure legitimate reviewer decision ... state diff: decision, approved_amount_cents
[PASS] object-level control (secure)     ... 404 read / 404 write, state unchanged
[PASS] object-level control (vulnerable) ... 404 read / 404 write, state unchanged

WALKTHROUGH PASSED — 8/8 cases met every expectation

Exit codes, observed:

Invocation Exit
default (full) 0
--mode secure (5/5 cases) 0
--mode nonsense 2
--target http://example.invalid 2

It only knows the two Compose service names, and there is no argument that redirects it or names a
different object. That it detects failure rather than always passing is proved in the suite: point
it at a service that cannot leak and the vulnerable cases fail; point the secure cases at the leaking
service and those fail.

Audit-event cardinality, and a trap avoided

The runner needs to assert "exactly one audit event" without reading a log file, so the demonstration
boundary now exposes the bounded, already-redacted audit history at /demo/events, and /demo/reset
clears it.

The first version counted a delta across that ring and passed in isolation while failing in the full
suite: the ring holds only rejection events, so once saturated the count is pinned at its maximum and
every delta reads zero — an assertion that could never fail. Cases now reset first and assert an
absolute count, and test_the_audit_history_stays_bounded pins the bound itself.

Regression matrix

tests/test_regression_matrix.py maps each FR-015 row to the named test that proves it and asserts
that test exists, so renaming or deleting a load-bearing test fails the gate instead of quietly
shrinking the matrix. A second test pins the required row set so a row cannot disappear either.

Verification

docker compose build --no-cache     # ~11s from a clean checkout
docker compose run --rm verify      # ~9s
--- ruff format --check    34 files already formatted
--- ruff check             All checks passed!
--- mypy --strict          Success: no issues found in 33 source files
--- pytest                 360 passed in 4.62s
--- verification complete

Well under the five-minute budget. The image now builds with uv sync --locked, so a lockfile that
drifts from pyproject.toml fails the build rather than silently installing something else — which
is how the stale lock from moving httpx into the runtime dependencies was caught.

Every documented manual command in the README was run literally against the running container: both
reads, the mixed-body rejection, the legitimate employee edit, the reviewer decision, the non-owner
control, /demo/state, /demo/events, and /demo/reset.

Documentation

The README now opens with the one command that shows everything, then explains BOLA versus BOPLA in a
table, both wrong data flows in a diagram, both halves of the fix (explicit response schema per
actor; explicit request schema plus assignment by name), and a short section on what property
authorization is not — client-side filtering, undocumented fields, hard-to-guess names, hidden form
fields, blocklists. It ends with the OWASP and CWE references. Every example uses only the fixed
fictional fixture, and the vulnerable service is marked intentionally unsafe and local-only
throughout.

Acceptance

Acceptance item Where it is proved
One command runs the whole walkthrough from fresh state and exits zero, printing all six observations per case run above; tests/test_walkthrough.py
Walkthrough proves vulnerable disclosure and the secure projection cases 1–2
Walkthrough proves vulnerable mutation, and the secure 400 with one audit event and unchanged state cases 3–4
Walkthrough proves both legitimate updates and identical non-owner behavior cases 5–8
The runner exits nonzero when an expectation is not met test_the_runner_fails_when_the_vulnerable_cases_cannot_hold, ..._secure_cases_...
Only enumerated modes accepted; no argument redirects it test_the_runner_rejects_anything_but_its_enumerated_modes, exit codes above
Every FR-015 row maps to a named test, checked mechanically tests/test_regression_matrix.py
docker compose run --rm verify stays the single gate, green from a clean checkout, well under five minutes run above
Actions runs that same command on push and pull request .github/workflows/verify.yml
README teaches the flaw, impact, fix, and preserved legitimate behavior from the fixed fixture README.md
Public text stays fictional, local-only, and marks the vulnerable service unsafe README.md

Notes

  • The walkthrough's own assertions run inside verify against both variants over real loopback HTTP,
    which is what keeps the single gate hermetic. The containerized walkthrough service is the same
    code, run against the two Compose services for a human to watch.
  • walkthrough uses an entrypoint rather than a command, so run --rm walkthrough --mode secure
    appends the flag instead of replacing the invocation.
  • /demo/events exposes only events that were already redacted before reaching the log; a test
    asserts it carries no property name, value, or credential.

Deliberately out of scope

Licensing, publication metadata, the exposure review, and any visibility change (SLICE-004).

Closes #5

Add the fixed walkthrough runner: it resets both services and proves every
required case — vulnerable disclosure, secure projection, vulnerable mass
assignment, secure whole-request rejection, the legitimate employee edit, the
legitimate reviewer decision, and the non-owner control in both variants —
printing the actor, both authorization verdicts, the HTTP outcome, the
authorized key set, and the state diff for each, and exiting nonzero the moment
one is missing. It accepts only its enumerated modes and knows only the Compose
service names, so no argument can point it elsewhere.

So it can assert audit-event cardinality without reading a log file, the
demonstration boundary now exposes the bounded, already-redacted audit history,
and reset clears it. Counting a delta across a full ring would silently read
zero, so cases reset first and assert an absolute count.

tests/test_regression_matrix.py maps every FR-015 row to the named test that
proves it and fails if one is renamed or deleted. GitHub Actions runs the same
`docker compose run --rm verify` the README documents, with no separate
pipeline. The README now explains BOLA versus BOPLA, both wrong data flows, both
halves of the fix, why client-side filtering and obscure property names are not
authorization, and how the suite proves security alongside preserved legitimate
behavior.

The image builds with `uv sync --locked` so a lockfile that drifts from
pyproject fails the build instead of installing something else.

Refs #5
@maximalfocus
maximalfocus merged commit b4c5d99 into main Aug 16, 2026
2 checks passed
@maximalfocus
maximalfocus deleted the issue/5-walkthrough-and-release branch August 16, 2026 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SLICE-003: integrated walkthrough, regression matrix, and local release

1 participant