Explore deterministic profiling gates for Crabbox runs

[zozo123]

Problem

Crabbox is good at running a repository's checks in disposable environments, but performance regression gates are still mostly outside its contract. Teams either rely on noisy wall-clock benchmarks on shared CI runners, maintain dedicated benchmarking machines, or skip perf gates because they are too flaky to trust.

This came up while reviewing the closed WASI provider PR (#211). A general WASI provider is probably not a fit for Crabbox's default workload: most repos need a shell, toolchains, subprocesses, sockets, and package managers. But deterministic profiling/perf gates may be a concrete capability where a restricted runtime is not a limitation; it is the feature.

Use case

A repo wants a CI check like:

crabbox run --profile perf -- go test ./parser -bench BenchmarkParse --bench-budget instructions=5000000

or, for wasm-compilable targets:

crabbox run --provider wasi --profile perf --fuel-budget 5000000 -- ./parser.test.wasm

The important outcome is not merely running WASI. The outcome is a reproducible run artifact that can say: this PR increased instruction count, allocations, CPU samples, or another agreed performance metric beyond budget.

Why Crabbox

Crabbox already owns the pieces that make this useful as a product surface:

• disposable execution environments;
• normalized run summaries and timing output;
• artifact capture/publishing;
• provider-specific execution without making callers learn each backend;
• policy-style gates that can fail CI.

A profiling gate would turn performance evidence into a first-class run result instead of a pile of provider-specific logs.

Possible shape

Start with a narrow, testable MVP rather than a broad profiler abstraction:

• Add a profiling/perf evidence mode for crabbox run that records bounded artifacts and a small JSON summary.
• Support one deterministic metric first, likely instruction/fuel counts for wasm-compilable test binaries where the number is stable enough to gate on.
• Treat WASI as an implementation detail or specialized target, not as a general replacement for Linux/container providers.
• Allow later backends to add native profilers (perf, pprof, Instruments, async-profiler, provider timing) behind the same evidence contract if there is demand.

Non-goals

• Reopen feat: add experimental --provider wasi delegated-run provider (wazero primary, wasmtime opt-in) #211 as a generic WASI provider without a concrete workload.
• Claim WASI can run arbitrary repo test suites. It cannot.
• Add a long-lived runtime dependency unless it proves a unique gate that current providers cannot offer.
• Build a general observability platform inside Crabbox.

Open questions

• Is deterministic instruction/fuel metering valuable enough to justify a first profiling gate?
• Which real repo/workload should be the proving case?
• Should this be modeled as a provider capability, a run evidence feature, or both?
• What artifact schema is small enough to review and stable enough for CI?

If the answer is yes, #211 can be treated as prior art for the runtime plumbing, but the product ask should be narrower: reproducible performance evidence and budget enforcement for Crabbox runs.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed July 31, 2026, 3:12 PM ET / 19:12 UTC.

Summary
Keep open: this remains the canonical product-direction question for deterministic Crabbox run evidence. Current main documents the intended contract but explicitly has no fuel-budget, WASI, or metering runtime, and the linked open design PR is documentation-only; a maintainer must choose the runtime and security boundary before implementation. Likely related people: steipete (high confidence) and vincentkoc (medium confidence).

Reproducibility: not applicable. This issue requests a new execution and evidence capability rather than reporting broken existing behavior.

Root-cause cluster
Relationship: canonical
Canonical: #280
Summary: This issue is the canonical product-direction record for deterministic evidence; related documentation work defines the boundary but does not implement the requested runtime or gate.

Members:

• partial_overlap: docs: define deterministic perf evidence #857 - Merged documentation establishes terminology and the evidence contract but does not provide metric collection or budget enforcement.
• partial_overlap: docs: design deterministic fuel evidence #1196 - The open documentation-only design refines a proposed workload and safety constraints while leaving implementation and approval decisions unresolved.
• superseded: Proposal: pure-Go WASM/WASI delegated-run backend (wazero) for untrusted single-process compute #533 - The issue discussion explicitly consolidated the broader embedded WASI-provider proposal into this narrower deterministic-evidence question.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.

Maintainer decision needed

Question	Recommendation
Should Crabbox sponsor a narrowly sandboxed deterministic-fuel MVP, and if so, which runtime distribution and execution boundary are acceptable for the first supported workload?	Approve a narrow external helper MVP: Use one pinned prebuilt-module workload with a subprocess meter, network denied by default, bounded filesystem and environment, plus a provider-neutral evidence schema.

Why: The checked-in contract deliberately leaves the helper distribution channel, metering scope, and runtime security boundary open; automation cannot safely choose those public and security-sensitive product commitments.

Next step
A maintainer must first sponsor a concrete runtime and security boundary; this is not a safe autonomous repair or feature-implementation candidate.

Security
Needs attention: The proposal is security-sensitive because its value depends on a new metered execution boundary that has not yet received a runtime or isolation decision.

Review details

Best possible solution:

Approve one narrow prebuilt-module fuel-metering MVP with a fixed proving workload, provider capability gate, backward-compatible timing evidence, and explicit deny-by-default runtime policy before adding CLI budget flags.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this issue requests a new execution and evidence capability rather than reporting broken existing behavior.

Is this the best way to solve the issue?

Unclear: the contract is a maintainable direction, but the first runtime, workload, and security boundary need explicit maintainer approval before it becomes the best implementation path.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• Choosing an embedded or subprocess WebAssembly runtime without a maintainer-approved filesystem, environment, network, timeout, and upgrade boundary could add a new execution security surface.
• An implementation before the workload and metric contract are approved could create incompatible budget semantics that CI users later depend on.

Codex review notes: model internal, reasoning high; reviewed against 5aa1c3983874.

Label changes

Label justifications:

• P3: This is a speculative, unimplemented capability proposal rather than a confirmed current-user regression.
• impact:security: The requested metered runtime would introduce an execution and sandboxing boundary that requires explicit security review.

Evidence reviewed

Security concerns:

• [medium] Define the metered runtime boundary before implementation — docs/features/deterministic-perf-evidence.md:147
  The proposed backend needs explicit limits for filesystem access, inherited environment, network, host imports, timeouts, and engine upgrades before any execution helper can become a supported Crabbox surface.
  Confidence: 0.96

What I checked:

• Current contract is documentation-only: The deterministic-performance document explicitly states that Crabbox does not yet ship a perf/fuel budget, WASI backend, or wasmtime-fuel backend; it describes the desired feature boundary rather than an implemented capability. (docs/features/deterministic-perf-evidence.md:8, 5aa1c3983874)
• Current timing JSON has no deterministic metric: TimingReport currently carries provider, lifecycle timings, command timing, artifacts, and normalized outcome fields, but no fuel/instruction result or budget verdict. (internal/cli/timing.go:9, 5aa1c3983874)
• Required implementation boundary remains unchosen: The checked-in contract requires a dedicated provider capability and a runtime that denies network and ambient environment by default, constrains filesystem access, and fails closed on incomplete provenance; those are product and security choices, not a ready bug fix. (docs/features/deterministic-perf-evidence.md:129, 5aa1c3983874)
• Merged documentation provenance: The linked merged documentation PR docs: define deterministic perf evidence #857 established the contract and explicitly described future metric budgets; it did not add runtime code or flags. (docs/features/deterministic-perf-evidence.md:1, 974d5c6ca1b8)
• Open design work does not implement the capability: The provided record for docs: design deterministic fuel evidence #1196 says it is documentation-only and leaves the helper distribution channel, metering scope, and versioning policy for maintainer approval. (4905737a0c5e)
• Maturity scorecard unavailable: The required maturity helper could not find qa/maturity-scores.yaml in this checkout, so no maturity:stable label can be justified.

Likely related people:

• steipete: Their July 1 comment identifies this as the canonical deterministic-evidence question and their open design PR is the current concrete proposal for its bounded runtime contract. (role: recent product-direction contributor; confidence: high; commits: 4905737a0c5e, ebf6923a7963; files: docs/features/deterministic-perf-evidence.md, internal/cli/timing.go)
• vincentkoc: They authored the merged documentation change that introduced the current deterministic-perf evidence contract and linked it to this issue. (role: merged design-contract author; confidence: medium; commits: 974d5c6ca1b8; files: docs/features/deterministic-perf-evidence.md, docs/observability.md, docs/source-map.md)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[zozo123]

@clawsweeper try again

[zozo123]

Deterministic profiling gates: feasibility, MVP design, and the four open questions

Heads-up: the automated ClawSweeper/Codex pass on this issue died with a transport-capacity error, so there's no real analysis on the thread yet. Below is a hand-done pass against the current tree (HEAD d3d1891, 0.31.1-dev). I went into the Provider/DelegatedRunBackend contract and the existing proof/timing plumbing to ground this.

First, a grounding fact that changes the framing: PR #211 (the general WASI provider) was reverted cleanly. There is no internal/providers/wasi/, and go.mod/go.sum carry no wazero/wasmtime/wasmer. So #280 is a greenfield question against today's contract, not a resurrection of #211. Good — because most of #211's 2,500-line bulk is exactly what the non-goals say we don't want.

---

The four open questions

(a) Is deterministic metering valuable enough for a first profiling gate? — Yes, but only as a determinism/regression gate, explicitly not a performance benchmark.

The value is the one property wall-clock can never have: bit-reproducibility across providers and CI runs. A fuel count from a wasm execution is identical on a Mac, a Hetzner box, and blacksmith, so it can be a hard CI assertion (fuel <= baseline) in a way commandMs never can. That is the first gate worth shipping: catch accidental algorithmic regressions and non-determinism, not "8% slower."

Sell it honestly as "a reproducible counter you can assert on," not as a benchmark. Fuel ignores cache behavior, SIMD, syscalls, and I/O — it is not a proxy for real-world speed. Under the regression framing it's clearly worth a v1; under a performance framing it would over-promise.

(b) Which proving workload? — A pure-compute, no-syscall, deterministic Rust micro-bench compiled to wasm32-wasip1.

In priority order:

1. blake3 hash-a-fixed-buffer loop (b3sum-style). Real OSS, cleanly wasm32, pure CPU, zero syscalls in the hot loop, and trivially parameterizable (hash a fixed N-byte vector K times) so fuel is a clean function of input size.
2. Fallback: a pure-Rust parser bench — serde_json over a committed JSON fixture, or pulldown-cmark over a fixed document — reading the fixture into memory before the metered region.

Avoid Go for the proving case despite the "pure-Go/Rust" framing: Go's wasm target drags in scheduler/GC and a heavier prologue, inflating fuel with non-workload work and adding determinism caveats. Keep Go as a later stress workload precisely because exercising the GC path is a good test of the determinism claim.

The bench must: (1) read input before the metered region, (2) do no I/O / clock / random inside it, (3) commit the exact input fixture and expected output hash so the gate asserts both correctness and fuel.

(c) Provider-capability, run-evidence, or both? — Both, asymmetrically: one thin new capability gate + extend the existing structured evidence record. Do not touch the Markdown proof.

crabbox already has two distinct mechanisms and #280 sits at their seam:

• Feature/FeatureSet (provider_backend.go:335-369) — a capability declaration; FeatureRunProof already rejects --emit-proof at preflight (provider_backend.go:1301) when a backend can't do it.
• Run evidence — the Markdown proof (renderRunProof, run_artifacts.go:310-396, reviewer-facing prose) and the TimingReport (timing.go:9, fully JSON-tagged, machine-readable, emitted under --timing-json).

Decision:

• Evidence (substance): add the metric to TimingReport, sitting next to the (non-deterministic) commandMs — the honest framing is "here is the deterministic counterpart to the noisy timer." Reuses --timing-json, the delegated delegatedTimingReport path, and every downstream JSON consumer for free.
• Capability (gate): add one Feature (e.g. FeatureRunFuel = "run-fuel") so any non-wasm provider (most of the fleet) is rejected at preflight, mirroring FeatureRunProof verbatim. Without this you'd emit a meaningless/absent number on Docker/AWS/Hetzner and silently break cross-provider comparability.
• Do NOT add it to the Markdown proof — that's a human narrative; a raw instruction count there is noise and widens a stable, hand-reviewed surface for no benefit.

(d) Artifact schema — small enough to review, stable enough for CI?

Two layers, because there are two distinct consumers: the timing record (always-on telemetry) and the gate evidence file (opt-in, for orchestrators). Both reuse existing serializers.

Telemetry — extend TimingReport (timing.go:9) with a pointer + omitempty so every existing consumer and golden test is byte-unchanged when fuel is absent:

// in TimingReport
Fuel *FuelReport `json:"fuel,omitempty"`

type FuelReport struct {
    Instructions  uint64 `json:"instructions"`   // the metered count — the assertable value
    Mechanism     string `json:"mechanism"`      // e.g. "wasmtime-fuel"
    Engine        string `json:"engine"`         // e.g. "wasmtime@<ver>" — pin for comparability
    Module        string `json:"module"`         // wasm module identity (path or sha256)
    Deterministic bool   `json:"deterministic"`  // engine config asserted no non-determinism sources
}

Engine/Mechanism/Module are mandatory provenance, not decoration: a bare integer is uncomparable: pinning the engine string is what lets a gate refuse to compare counts from different metering stacks (the flakiness failure mode). No per-phase breakdown in v1.

---

Concrete MVP proposal

CLI surface — reconcile --profile perf / --bench-budget / --fuel-budget into one design

Two naming facts from the tree drive this:

• --profile is already taken — it's the global workspace profile flag (cfg.Profile; controller.go:36, profiles.go:27-31, config.go:5625). --profile perf would collide. Reject it.
• --require-artifact is the gate to mirror — it's its own gate flag (not an overload), runs after the command, and returns exit(7, …) when unmet (run_artifacts.go:35-49).

So: separate what to measure (metric, provider-declared) from the budget (the threshold), in one flag namespace:

crabbox run --perf-budget <metric>=<limit> [--perf-evidence <path>] -- <command...>

• --perf-budget metric=limit — repeatable (mirrors --require-artifact's stringListFlag, run.go:231). limit is an unsigned inclusive max. This collapses both issue forms into one flag — the metric name carries the provider semantics, so no per-provider flag proliferation:
  • WASI/fuel path: --perf-budget fuel=5000000
  • bench/instructions path: --perf-budget instructions=5000000
  • Validate the metric name against what the backend declares, gated by FeatureRunFuel (mirror the FeatureRunSession guard at run.go:476-479).
• --perf-evidence <path> — optional; writes the evidence JSON (exact analog of --lease-output, run.go:217). Preflight via preflightLocalOutputPath("perf evidence", path, false) (run.go:331) and reject collision with other output paths (run.go:337-342).

Drop --provider wasi from the user-facing surface: --provider already exists globally, and WASI stays an implementation detail of how the fuel metric is produced, not a general execution provider. The gate runs whether or not --perf-evidence is passed — the budget is the gate, the file is the record (same split as --require-artifact vs --download).

Perf-evidence JSON schema (the opt-in gate file)

Flat, camelCase, omitempty on optionals — same shape rules as RunSessionHandle (provider_backend.go:822-831), written via the shared writeJSONFile (artifacts.go:619) inside a writePerfEvidence helper modeled on writeRunLeaseOutput (run.go:1555-1563). Wrap a []PerfEvidence so multiple budgets stay one object:

{
  "provider": "wasi",
  "leaseId": "cbx_3f9a1c2d4e5b",
  "runId": "run-20260614-001",
  "engine": "wasmtime@<ver>",
  "results": [
    {
      "metric": "fuel",
      "measured": 5821447,
      "budget": 5000000,
      "exceeded": true,
      "unit": "fuel"
    }
  ],
  "exceeded": true
}

The per-result measured/budget/exceeded are the assertable fields; the top-level engine is the provenance that makes cross-run comparison safe; the top-level exceeded is the one-glance gate result for orchestrators.

Gate semantics — budget exceed → nonzero exit

Mirror the --require-artifact gate (run_artifacts.go:35-49) exactly:

1. Run after the command. (The command's own nonzero exit already flows through recordFailure → ExitError{Code}, run.go:1474-1486.)

2. Per metric, exceeded = measured > budget — inclusive limit, so == passes (deterministic, matches "budget" intuition).

3. If any metric is exceeded: write the evidence file first (so the artifact records the failure), then:

   return recordFailure(exit(7, "perf budget exceeded: %s measured=%d budget=%d", metric, measured, budget))

   Exit 7, identical to the require-artifact gate — crabbox's established post-run gate/verification code. The run summary already prints exit=%d (run.go:1922-1929), so it surfaces there too.

4. Stderr breadcrumb in the existing style (delegated_downloads.go:54):
   perf budget fuel measured=5821447 budget=5000000 exceeded=1

5. Unsupported providers: guard with exit(2, "--perf-budget is not supported for provider=%s yet", …), mirroring the lease-output guard (run.go:476-479) and the delegated require-artifact guard (provider_backend.go:1289).

Clean code-split, all already conventions in this file: 2 = bad invocation, 7 = ran fine but a gate failed, command's own code = the workload itself failed.

---

Honest limits (gate constraints, not footnotes)

Fuel is deterministic by construction — charged per guest wasm instruction, independent of wall-clock, CPU, or scheduler. But the soundness is narrow, and these must ship as documented constraints:

• Host/import calls cost zero fuel. Fuel meters only guest instructions. Every WASI syscall (fd_read, clock_time_get, random_get, poll_oneoff) is invisible to the counter. A regression that moves work into host I/O won't register. The gate measures compute in the guest, not total cost.
• Nondeterministic imports poison reproducibility. If the module reads the clock, randomness, or does data-dependent I/O, the instruction path itself can branch run-to-run. The gate is only sound for pure, input-fixed, deterministic modules — hence the fixed-corpus workload in (b) and a /work-style sandbox with explicit --allow-env.
• Engine-version sensitivity. A wasmtime upgrade or a different wasm32-wasip1 compiler can shift the absolute count. Pin both toolchain and engine; treat the budget as comparable only within a fixed (compiler, engine) pair — a relative-within-a-pinned-stack gate, not an absolute cross-version one. (This is why engine is mandatory in the schema.)

One sharp risk in the issue's own examples: wazero cannot deliver the headline number. The issue/PR #211 named "wazero primary, wasmtime opt-in," but wazero exposes no fuel/gas API — only wasmtime has real per-instruction fuel metering (Config::consume_fuel, Store::set_fuel/get_fuel). The embedded zero-dep story and the deterministic-metric story are mutually exclusive. The MVP must pin wasmtime as the metering engine (or drop the fuel metric). The --fuel-budget 5000000 example is only deliverable on the wasmtime path.

What's reuse vs genuinely new

• Reuse (the evidence/gate half is cheap): RunRequest already carries EmitProof/ProofTemplate/ArtifactGlobs/RequiredArtifactGlobs/TimingJSON (provider_backend.go:749-753); RunResult already returns ExitCode/durations/Artifacts (provider_backend.go:795-808). The DelegatedRunBackend seam (Warmup/Run/List/Status/Stop, provider_backend.go:113-120) is the right home for a thin metering backend.
• Genuinely new (bounded): one FuelReport field on TimingReport, one FeatureRunFuel capability + preflight reject, one --perf-budget/--perf-evidence flag pair with exit-7 gate, and a small wasmtime-fuel backend (take a prebuilt .wasm, run under consume_fuel, read get_fuel, compare, emit JSON — a few hundred lines).
• Explicitly drop feat: add experimental --provider wasi delegated-run provider (wazero primary, wasmtime opt-in) #211's general-provider bulk — host builtins (ls/cat/echo), general guest filesystem, manifest sync, symlink-escape resolution, shell. Those existed because feat: add experimental --provider wasi delegated-run provider (wazero primary, wasmtime opt-in) #211 tried to be a general execution environment, which Explore deterministic profiling gates for Crabbox runs #280 rejects as a non-goal. The profiling target is a thin metering backend, not a re-skinned feat: add experimental --provider wasi delegated-run provider (wazero primary, wasmtime opt-in) #211.

Bottom line: worth doing as a narrow determinism/regression gate; feasible today on a wasmtime-fuel path (not wazero) for pure, input-fixed wasm targets with pinned toolchain+engine. The existing proof/timing/artifact plumbing makes the evidence and gate half cheap; the genuinely new work is a small fuel backend plus a TimingReport field, a capability gate, and the --perf-budget/--perf-evidence flag pair.

[steipete]

@clawsweeper re-review

The prior advisory failed during transport. The issue now includes a current-main design analysis; please refresh the advisory.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/27532184676
• Updated: 2026-06-15T08:05:46.188Z

[steipete]

Keeping this as the canonical product question for deterministic Crabbox run evidence. Current main has timing JSON, run evidence, artifact gates, and delegated module execution, but still has no deterministic metric or budget contract.

Consolidating #533 here. That proposal is a broader embedded WASI execution backend with a distinct in-process security boundary; its current prototype does not establish the fuel/instruction-budget contract requested here.

Before implementation, this still needs one real Crabbox workload, one metric, a provider-neutral evidence schema, and an explicit runtime/security decision.