Security enforcement, agent hardening, and a benchmark harness

.gitignore

@@ -14,3 +14,7 @@ dist/
 
 # Local README preview render
 .readme-preview.html
+
+# Benchmark runs: reports are artifacts of a machine, a model and a moment.
+# A published score belongs in BENCHMARKS.md, with the command that produced it.
+.cude-bench/

BENCHMARKS.md

@@ -0,0 +1,142 @@
+# Benchmarks
+
+## Where Cude Code actually stands
+
+**Cude Code has no verified score on Terminal-Bench, SWE-bench Verified, or any
+other independent leaderboard.** Nothing in this repository claims otherwise,
+and no number appears in the README that has not been produced by a run anyone
+can repeat.
+
+That gap is real, and this file is about closing it properly rather than
+papering over it. A benchmark figure is worth exactly as much as the harness
+behind it: who graded it, on what dataset, at what version, and can someone
+else get the same number. Until Cude has been through an official evaluator,
+the honest statement is the one above.
+
+What exists now is the machinery that produces such a number — a harness that
+runs the real agent against real tasks and grades it by something other than
+the model's own claim of success.
+
+```bash
+cude bench list                 # what can be run, and what each one needs
+cude bench local                # Cude's own suite: no Docker, no dataset, no network
+cude bench swebench --dataset swe-bench-verified.jsonl --limit 25
+cude bench terminal-bench --tasks path/to/terminal-bench/tasks
+```
+
+## How grading works
+
+Three rules the harness enforces, because they are the ones that make a number
+mean anything:
+
+1. **The grader is not the agent.** Every task is graded by a command run after
+   the agent has stopped, in the task's own directory, through the shell — not
+   through the agent's tools. `TASK COMPLETE:` in a model's final message has
+   no effect on the result. A run where the model declares victory and the
+   tests still fail is a failure.
+2. **Every task starts out failing.** There is a test asserting this for the
+   whole local suite: a task whose verifier passes before the agent touches
+   anything measures nothing.
+3. **Every run states its provenance.** A run is labelled `local`,
+   `unofficial`, or `official`, and the report repeats the caveat above the
+   number. Only a grade produced by a dataset's own evaluator is written
+   without one.
+
+Each task runs in its own temporary directory with the workspace root pointed
+at it, so a task cannot reach the machine or another task — there is a test
+that tries to write outside the sandbox and asserts it fails. Tasks run one at
+a time, deliberately: the workspace root and the process working directory are
+global, so overlapping tasks would mean one task's shell commands executing in
+another's tree.
+
+## The local suite
+
+Eight tasks, graded by `node --test`, which is present wherever Cude runs.
+No Docker, no dataset download, no network — the suite you can run on every
+change.
+
+| Task | What it exercises |
+| --- | --- |
+| `implement-fizzbuzz` | Write a module so an existing test passes |
+| `fix-slugify` | Three real bugs, found by reading a failing test |
+| `fix-divide-by-zero` | An edge case the implementation never handled |
+| `implement-arg-parser` | Implement against a spec that exists only as tests |
+| `implement-retry` | Async control flow, including the give-up path |
+| `multi-file-rename` | A rename across three files without breaking imports |
+| `patch-precise-edit` | Change one function and leave its neighbours alone |
+| `document-module` | Read code and write accurate prose about it |
+
+Tasks whose grade depends on a test file restore that file before grading, so
+deleting or editing the test cannot pass a task.
+
+## SWE-bench Verified
+
+This harness does **not** grade SWE-bench. Grading requires the official Docker
+evaluation images, and a score produced any other way is not the score people
+mean when they quote one.
+
+What `cude bench swebench` does is the half that is Cude's job: check out each
+instance at its base commit, run the agent on the issue text, and write the
+working-tree diff into a `predictions.jsonl` in the format the official
+evaluator consumes.
+
+```bash
+# 1. Get the dataset
+#    https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified
+
+# 2. Produce predictions
+cude bench swebench --dataset swe-bench-verified.jsonl \
+  --provider anthropic --model claude-sonnet-5 --limit 25
+
+# 3. Grade them with the official harness
+python -m swebench.harness.run_evaluation \
+  --predictions_path .cude-bench/predictions.jsonl \
+  --dataset_name princeton-nlp/SWE-bench_Verified \
+  --run_id cude
+```
+
+The number that comes back from step 3 is a number worth publishing. When one
+exists, it goes here with the model, the date, the Cude version and the run id
+next to it.
+
+## Terminal-Bench
+
+Terminal-Bench grades inside containers it builds itself, driven by its own
+`tb` runner. `cude bench terminal-bench` reads a task directory, hands the
+instruction to the agent in a sandbox, and runs whatever test script the task
+ships. That is useful during development and it is **not** a Terminal-Bench
+score; runs are labelled `unofficial` and the report says so.
+
+For a quotable number, run the official harness with Cude as the agent under
+test.
+
+## What the harness measures besides pass rate
+
+Every run records what the loop had to do to get there, because a pass rate on
+its own does not tell you what to fix:
+
+- **tool calls** and **tool errors** — how much work each task took, and how
+  much of it was wasted
+- **repaired calls** — calls whose name or arguments had to be corrected
+- **compactions** — turns where the conversation had to be compacted to stay
+  inside the context window
+- **stop reason** — `completed`, `max_iterations`, `verification_failed`,
+  `budget_exceeded`, `timeout`
+- cost, tokens and wall-clock time per task
+
+Reports are written to `.cude-bench/<suite>-<timestamp>/` as `run.json` and
+`report.md`.
+
+## Reproducing a run
+
+```bash
+git clone https://github.com/Emrevrg/Cude-Code.git
+cd Cude-Code && npm install && npm run build
+cude config set-key anthropic <key>
+cude bench local --provider anthropic --model claude-sonnet-5
+```
+
+The harness itself is covered by the test suite (`test/bench.test.mjs`), which
+drives the real agent loop against a scripted local server — sandboxing,
+grading, the report, and the refusal to accept an unverified completion are all
+exercised without an API key.

CHANGELOG.md

@@ -4,6 +4,112 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+### Benchmarking
+
+Cude has no verified score on any independent leaderboard, and this release
+does not invent one. It adds the harness that can produce one — `cude bench` —
+together with the agent-loop work that a long benchmark run needs in order to
+finish at all.
+
+- **`cude bench local|swebench|terminal-bench|list`.** Each task runs in its own
+  temp sandbox with the workspace root pointed at it, and is graded by a shell
+  command run *after* the agent stops — the model's "TASK COMPLETE:" has no
+  bearing on the result. Reports (`run.json`, `report.md`) carry a provenance
+  label: `local`, `unofficial` or `official`, with the caveat printed above the
+  number.
+- **The local suite** — eight tasks graded by `node --test`: implement against
+  a test, fix real bugs, rename across files, make a precise single-function
+  edit, document a module. No Docker, no dataset, no network. A test asserts
+  every task fails before the agent touches it, and tasks graded by a test file
+  restore that file first, so deleting the test cannot pass a task.
+- **SWE-bench Verified** — checks out each instance at its base commit, runs the
+  agent, and writes `predictions.jsonl` for the *official* Docker evaluator.
+  Cude does not grade itself on it. See [BENCHMARKS.md](BENCHMARKS.md).
+
+### Agent
+
+Six changes to the loop, each of which the benchmark harness measures (O1–O6):
+
+- **Context compaction.** The loop re-sends the whole conversation every turn,
+  so a run that read a few large files did not degrade — it died on a
+  context-window error. Old tool results are digested, then whole steps are
+  dropped, oldest first, with a note left where they were. The turn-sequence
+  invariant is preserved at every budget, which is tested.
+- **Tool-call repair.** `writeFile` → `write_file`, `file_path` → `path`, `bash`
+  → `run_command`, a JSON object inside a markdown fence → arguments. Only
+  unambiguous corrections are applied, and every one is reported. A misnamed
+  call used to cost a full iteration.
+- **Parallel reads.** A turn whose calls are all read-only runs concurrently;
+  anything that mutates forces sequential execution.
+- **`apply_patch` is atomic.** It located hunks by line number and skipped a
+  `-` line that did not match *while still inserting the `+` lines around it* —
+  a corrupted file, reported as success. Hunks are now found by content, all of
+  them apply or none do, and the error names the hunk that failed.
+- **Verification before completion.** `verifyCommand` runs the project's own
+  tests when the model says it is finished; a failure is handed back with its
+  output and the loop continues. A run that never satisfies it stops with
+  `verification_failed` instead of `completed`.
+- **Retry with backoff.** 429 and 5xx responses are retried with exponential
+  backoff and jitter, honouring `Retry-After`. One rate limit used to end a
+  run.
+
+Also fixed: child processes inherited `NODE_TEST_CONTEXT`, so any nested
+`node --test` reported success regardless of its tests — a verification command
+that always passes is worse than none.
+
+### Security
+
+A security core (`src/core/security.ts`) that every tool call now passes
+through, plus `cude security scan|audit|log|check`. The controls are enforced
+in code, not asked for in the system prompt, because the model is a confused
+deputy and not an adversary: it reads web pages, dependency READMEs and MCP
+results that anyone can write. Nine classes of exposure closed (S1–S9):
+
+- **S1 — Credential files are refused.** `.env`, `~/.ssh`, `~/.aws`, `~/.gnupg`,
+  `*.pem`, `*.key`, `.npmrc`, `.netrc`, service-account JSON and the rest are
+  unreadable through `read_file`, `grep_search`, `diff_files`, `copy_file`,
+  `get_file_info`, RAG indexing, Claw's `@path` mentions and `file://` URLs.
+  `.env.example` and other templates stay readable. RAG's file walk had gone
+  out of its way to include `.env` — the one dotfile it skipped the dotfile
+  rule for was the one holding the keys.
+- **S2 — Secrets are redacted before they leave.** All tool output passes one
+  choke point; anything matching a live credential shape becomes
+  `[CUDE:REDACTED:<rule>]` before it reaches the model, the terminal or a
+  session file. Placeholders and low-entropy values are left alone.
+- **S3 — Redaction markers cannot be written back.** `write_file`,
+  `replace_in_file` and `apply_patch` refuse content containing a marker, so a
+  placeholder can never overwrite the real value. Writing a *new* live
+  credential into a file asks first.
+- **S4 — Command analysis replaces the blocklist.** Three verdicts instead of
+  one boolean. Blocked outright: encoded PowerShell, base64-into-a-shell, and
+  commands that read credential material and send it over the network.
+  Confirmed: destructive commands, uploads, inline interpreter one-liners,
+  persistence, broad permission grants, reverse shells. A command's working
+  directory is now confined to the workspace root, and output is capped.
+- **S5 — Child processes no longer inherit credentials.** `run_command`,
+  `git_command`, `npm_command` and stdio MCP servers get an environment with
+  every credential-shaped variable removed. A malicious `postinstall` script
+  used to receive every API key the user had exported.
+- **S6 — Egress control.** Cloud metadata endpoints are always refused; only
+  `http`, `https` and `file` schemes are allowed; `file://` obeys the read
+  deny-list. `browser_screenshot` was the one write path in the tool set that
+  never checked the workspace boundary — it does now.
+- **S7 — Untrusted content is labelled.** Browser and MCP output is wrapped in
+  `<untrusted source="…">` and scanned for injection markers.
+- **S8 — Owner-only storage and an audit log.** `~/.cude` and everything in it
+  is written `0600`/`0700` on POSIX; session transcripts are redacted before
+  they are saved; every tool call is appended to `~/.cude/audit.log` with
+  redacted arguments and its outcome. Sessions also stopped ignoring
+  `CUDE_HOME`, which they had been writing around.
+- **S9 — `cude security scan`.** The same detection, pointed at a project: it
+  finds hardcoded credentials in source, reports credential files that git is
+  tracking, and exits non-zero under `--strict` for CI.
+
+Every control has a documented escape hatch — `CUDE_ALLOW_SECRET_FILES`,
+`CUDE_NO_REDACT`, `CUDE_ALLOW_UNSAFE_COMMANDS`, `CUDE_INHERIT_SECRETS`,
+`CUDE_AUDIT=0` — and `cude security audit` reports any that are set. See
+[SECURITY.md](SECURITY.md).
+
 ### New Features
 
 - **Cude Claw** (`cude claw`) — an interactive agent session that keeps context

README.md

@@ -33,6 +33,8 @@ cude chat
 - **Autonomous Agent**: Solve complex tasks with tool-use
 - **Cost Tracking**: Monitor spending, set budgets, get alerts
 - **Session Management**: Save and restore conversations
+- **Credentials Never Leave**: Key files are unreadable to the agent, secrets are
+  redacted out of every tool result, and child processes don't inherit your keys
 - **Privacy First**: Everything stays on your machine
 - **Pure CLI**: No Electron, lightweight and fast
 
@@ -179,6 +181,62 @@ Servers are verified before they are saved, tools are namespaced
 `mcp__<server>__<tool>` so none can shadow a built-in, and a server that fails
 to start is reported and skipped rather than taking the run down.
 
+### Benchmarks
+
+Cude Code has **no verified score on any independent leaderboard** — not
+Terminal-Bench, not SWE-bench Verified. What it has is the harness that
+produces one, so the claim can be checked rather than asserted.
+
+```bash
+cude bench list                 # suites, and what each one needs
+cude bench local                # 8 tasks graded by node --test — no Docker, no network
+cude bench swebench --dataset swe-bench-verified.jsonl --limit 25
+```
+
+Grading is independent of the agent: a verifier is a shell command run after
+the agent stops, so `TASK COMPLETE:` in the model's last message counts for
+nothing. Every task starts out failing (there is a test asserting it), every
+task runs in its own sandbox, and every report states whether it is a local
+run, an unofficial dataset run, or an official evaluator's grade. For
+SWE-bench, Cude emits `predictions.jsonl` for the official Docker harness to
+score — it does not grade itself.
+
+See [BENCHMARKS.md](BENCHMARKS.md) for the full method and the exact commands.
+
+### Security
+
+An AI agent holds your shell, your files and your API keys, and it reads
+content anyone can write — web pages, dependency READMEs, MCP results. Cude
+assumes that content is hostile and enforces the boundary in code rather than
+in the prompt.
+
+```bash
+cude security audit          # key storage, permissions, MCP trust, what is off
+cude security scan           # find hardcoded credentials in this project
+cude security scan --strict  # exits non-zero on a finding — for CI
+cude security check .env     # why a path is or is not readable
+cude security log            # every tool call the agent has made
+```
+
+What that buys you, without any configuration:
+
+- **Key files are unreadable.** `.env`, `~/.ssh`, `~/.aws`, `*.pem`, `.npmrc`
+  and the rest are refused by every read path — `read_file`, `grep_search`,
+  RAG indexing, `@path` mentions, `file://` URLs. `.env.example` still works.
+- **Secrets are redacted on the way out.** Anything key-shaped in a tool result
+  is replaced with `[CUDE:REDACTED:…]` before the model, the terminal or the
+  session file ever sees it — and a write that would put that marker back over
+  the real value is refused.
+- **Your keys stay out of child processes.** `npm install`, `run_command` and
+  MCP servers each get a scrubbed environment.
+- **Exfiltration is blocked, not confirmed.** A command that reads credential
+  material and sends it over the network, or an encoded PowerShell payload, is
+  refused outright. Cloud metadata endpoints are always unreachable.
+- **Everything is logged.** `~/.cude/audit.log`, redacted, append-only.
+
+Every control has a documented escape hatch, and `cude security audit` reports
+any that are switched off. See [SECURITY.md](SECURITY.md) for the full model.
+
 ### Autonomous Tasks
 ```bash
 # Code generation