Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,27 @@ jobs:
run: |
chock check
chock sync --repo . --check

# A stale generated figure fails the build: regenerate every docs/figures/*.svg and diff against
# what is committed. make_row_basis.py and make_social_card.py both shell out to the README's
# own quickstart block (see tools/quickstart_block.py) to draw their numbers from a real,
# freshly produced report.json, so context-report itself must be installed first.
figures:
name: Generated figures are not stale
runs-on: ubuntu-latest
# Reads the checkout; writes nothing (the regenerated files are diffed, never pushed).
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.12"
- run: pip install --require-hashes -r requirements/ci.txt
- run: pip install --no-deps -e .
- name: Regenerate figures
run: cd docs/figures && for g in make_*.py; do python "$g"; done
- name: Fail on drift
run: git diff --exit-code -- docs/figures
18 changes: 17 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,14 @@ it). Three real rows from the `report.json` this exact block just produced:
relative path `./my-plugin` you passed, so the hook only starts from the one cwd where that path
still points at the plugin — the exact failure mode the measurement below calls out.

The full statement has 11 rows, not 3 — every row's basis, from this exact quickstart block run
fresh:

<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/open-coder-ai/context-report/main/docs/figures/fig-row-basis-dark.svg">
<img alt="Of 11 rows in the report.json this quickstart block produces, 3 are recomputed from the artifact, 1 is the author's unverified claim, and 7 have not been measured this run — shown as absence of evidence, not a weak score." src="https://raw.githubusercontent.com/open-coder-ai/context-report/main/docs/figures/fig-row-basis-light.svg" width="760">
</picture>

Beyond one statement at a time, `context-report run` drives a whole manifest — subjects × models ×
tasks — and `context-report compare` puts every run of that manifest side by side; see
[`docs/cli.md`](docs/cli.md) for the manifest schema, `--dry-run`/`--n`/`--resume`, and which
Expand Down Expand Up @@ -143,7 +151,10 @@ skills. Three findings from that run:
`FAILED, 0 of 4`, exit 126. Every hook that runs, across both samples, allows on malformed input.
See [§5.2](paper/context-report.md#52-top-n-catalog-plugins).

![Eighteen plugins by four measured attributes](paper/figures/fig-catalog-status.svg)
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/open-coder-ai/context-report/main/docs/figures/fig-catalog-status-dark.svg">
<img alt="Grid of 18 catalog plugins by 4 measured attributes: three plugins fail reachability outright, every plugin that measures malformed-input handling passes it, and not-applicable cells are shown as a neutral dash, never a low score." src="https://raw.githubusercontent.com/open-coder-ai/context-report/main/docs/figures/fig-catalog-status-light.svg" width="760">
</picture>

**Cost spans two orders of magnitude.** Hooks that shell out to `npx` cost 916.8–941.5 ms p50; a
local script costs 7.3–53.9 ms. Context weight varies about a hundredfold across the sample,
Expand Down Expand Up @@ -244,6 +255,11 @@ merge either way.

## Part of open-coder-ai

<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/open-coder-ai/context-report/main/docs/figures/family-dark.svg">
<img alt="The open-coder-ai family: agentseam is the foundation, chock sits on it, chock-catalog feeds chock and generates the four plugin repositories, chock-threat-intel feeds the catalog, and context-report runs as a verification arm measuring all four." src="https://raw.githubusercontent.com/open-coder-ai/context-report/main/docs/figures/family-light.svg" width="800">
</picture>

| | |
|---|---|
| [agentseam](https://github.com/open-coder-ai/agentseam) | the primitives — one handler API and a verified capability matrix across 16 agents |
Expand Down
46 changes: 46 additions & 0 deletions docs/figures/family-dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
46 changes: 46 additions & 0 deletions docs/figures/family-light.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading