assay

A guardrailed API testing CLI for coding agents. You assay an API the way a lab assays a sample: under controlled conditions, with bounded exposure, to produce a recorded result you can trust. Point it at an OpenAPI spec (or just a base URL) and it exercises operations behind hard safety rails, validates the responses, and compiles structured findings.

Every request is host-locked, mutations are off unless you opt in, credentials are redacted from all output, and a per-run request budget and rate limit are enforced across process invocations. The agent drives; assay keeps it inside the authorized envelope.

Why Rust

assay is spawned by an agent, once per command, and must never touch anything outside its authorized scope. That rewards:

• No runtime to ship. One static binary, near-instant cold start, low resident memory.
• State that survives concurrency. Budget and rate limits live on disk and are guarded by a cross-process file lock, so two concurrent invocations can't both overspend.
• Types that make illegal states unrepresentable. Severity is an ordered enum where a >= b is the "at or above threshold" test — no rank table to drift out of sync.

Dependencies: chosen to not break

For a binary, the real guarantee of "never breaks" is a committed Cargo.lock — the build is reproducible forever, regardless of what happens upstream. On top of that:

• Everything load-bearing is 1.0+ and battle-tested: serde, serde_json, regex, url, uuid, sha2, clap, indexmap, chrono.
• YAML parsing uses serde_yaml_ng, a maintained, actively-supported parser.
• The two pre-1.0 crates whose APIs still move — ureq (blocking HTTP, no async runtime, small memory) and jsonschema (draft 2020-12 / draft-07 validation) — are pinned to exact versions, so an upgrade is a deliberate, reviewed act.

Architecture

cli            clap surface + the one place a Result becomes stdout/exit code
commands*      one handler per subcommand
http_executor  the single guarded request path: rails, auth, verdict, evidence
spec / spec_loader   OpenAPI model + loading, $ref deref, Swagger-2 fold-forward
verdict        is this response a bug? (status / schema / content-type)
sweep / fuzz   the two deterministic engines that produce findings
findings       typed findings, content-addressed ids, dedup, lifecycle
coverage / report    what got tested, and the CI gate
session_store / session_vars   .assay/ run state, evidence log, file lock
project_config      assay.json shape, validation, policy resolution
redaction / glob / json_path / urls / tri   focused, testable primitives

Every request funnels through http_executor::execute_call, the only code that talks to the network — so the guardrails can't be bypassed.

Design notes

• Findings are typed. Severity, Category, Source, Status are enums; severity ordering is the enum's own Ord.
• One error type. AssayError carries a code, a hint, and an optional upstream status; rendering it to a message / JSON envelope / exit code is a pure function of the value.
• HTTP method stays a validated string — checked at every entry point and only ever flowing through as lowercase. Enum-ifying it would be ceremony without safety gain.
• Schema validity drives findings and is faithful; the human wording of a schema error is best-effort evidence, not a contract.

Layout on disk

• assay.json — project config (base URL, headers, auth profiles, policy).
• .assay/ — per-run state, the cached spec, the request evidence log, findings, and session variables. Owner-only (0600/0700); add it to .gitignore.

Build, test, run

cargo build --release     # -> target/release/assay (single static binary)
cargo test                # unit tests for the pure logic
cargo clippy              # clean, no warnings

assay init <spec> --base-url <url>
assay endpoints --json
assay sweep --dry-run --json
assay call GET /users --json
assay report --ci

CI/CD

assay report --ci returns a deterministic exit code (0 pass, 1 findings, 3 incomplete, 2 tool error), so it drops into any pipeline as a read-only API contract gate against a staging deploy. See docs/ci.md for a ready-to-paste .gitlab-ci.yml, GitHub Actions + generic-shell equivalents, and the best-practice checklist.