Add box observability and resource controls

Makefile

@@ -8,6 +8,7 @@ syntax:
 	bash -n $(SHELL_FILES)
 	sh -n scripts/install.sh
 	python3 -c 'compile(open("guest/hermes-state", encoding="utf-8").read(), "guest/hermes-state", "exec")'
+	python3 -c 'compile(open("guest/tx9-logs", encoding="utf-8").read(), "guest/tx9-logs", "exec")'
 
 lint:
 	command -v shellcheck >/dev/null || { echo "shellcheck is required for make check" >&2; exit 1; }

README.md

@@ -3,8 +3,9 @@
 Isolated, portable agent boxes with **Claude Code, Codex, Hermes, and
 Executor**, running on Docker. Each box is a pair of containers on a private
 network — a free-rein agent container and an isolated Executor container —
-with all durable state on a named volume that travels as an encrypted,
-validated backup archive.
+with portable agent state on a named volume that travels as an encrypted,
+validated backup archive. Executor scratch state and runtime logs remain on a
+separate durable-but-non-portable volume.
 
 **Status: experiment, mid-transition.** The bash prototypes (smolvm `./box`
 and the compose `./docker/boxd`) have been retired in favor of a Go CLI
@@ -29,7 +30,7 @@ tx9 create                      (Go CLI, embedded build assets)
  │  │ agent ("linux land")  │      │ executor                │  │
  │  │ claude·codex·hermes   │      │ Executor daemon only    │  │
  │  │ gateway, sudo, free   │      │ dashboard / + MCP /mcp  │  │
- │  │ agent volume = the box│      │ scratch volume          │  │
+ │  │ agent volume = the box│      │ scratch + runtime logs  │  │
  │  │ no published ports    │      │ one configurable port   │  │
  │  └───────────┬───────────┘      └───┬─────────────────────┘  │
  │              │ http://executor:4788 │ bearer-token gated     │
@@ -44,6 +45,10 @@ layers ("upgrade" = rebuild image + recreate containers). Executor state is
 scratch and does not travel. Networking is always enabled because the agent
 tools require internet access.
 
+Run `tx9` with no arguments for an ASCII overview of every configured box,
+including status, CPU/RAM limits, volume usage/budgets, image version, and
+dashboard URL. The compact `tx9 list` table remains available for scripts.
+
 External host storage can be attached below `/mnt` without folding it into the
 portable box volume:
 
@@ -79,7 +84,7 @@ hb gateway-enable --confirm-single-writer I_CONFIRM_NO_OTHER_GATEWAY_USES_THIS_I
 hb gateway-disable
 hb verify-state
 hb wire-mcp        # (re)register Executor's authenticated HTTP MCP for claude+codex
-hb logs executor
+hb logs hermes
 ```
 
 The host CLI exposes the safe gateway lifecycle directly:
@@ -90,6 +95,52 @@ tx9 gateway enable <box> --confirm-single-writer
 tx9 gateway disable <box>
 ```
 
+## Logs and resource allocation
+
+Runtime output from the agent supervisor, Hermes gateway, and Executor is
+written to the component's own durable volume as rotating structured JSONL
+and readable text logs. Codex and Claude Code already keep native JSONL
+session histories on the agent volume; Hermes keeps its canonical SQLite
+history there. Query all of them from the host without weakening the
+agent/Executor filesystem boundary:
+
+```bash
+tx9 logs media-bot --source executor,codex --since 24h --grep failed
+tx9 logs media-bot --json | jq .
+tx9 logs export media-bot --output media-bot-logs.tar.gz
+```
+
+Known bearer-token forms are redacted by default. Log exports are created
+mode `0600`; they can still contain prompts, tool output, and private work, so
+treat them as sensitive. Existing boxes begin collecting the new durable
+runtime streams after `tx9 upgrade <box>`.
+
+TX9 captures all runtime output produced by the processes it owns and queries
+the native histories those agents persist. This is not an independent audit
+log for Executor calls that Executor itself never emits; operations initiated
+by an external dashboard/client may only be visible when upstream writes a
+corresponding runtime event.
+
+Container CPU and memory limits can be inspected, changed live, reset to the
+current defaults, and retained across upgrades or mount-driven recreation:
+
+```bash
+tx9 resources media-bot
+tx9 resources set media-bot --agent-cpus 6 --agent-memory 12GiB \
+  --executor-cpus 3 --executor-memory 4GiB \
+  --agent-volume-budget 96GiB
+tx9 resources reset media-bot
+```
+
+Docker's ordinary local named volumes have no portable, resizable hard quota.
+TX9 therefore reports their real used bytes and `unlimited` capacity by
+default; optional volume values are explicitly advisory budgets used for
+visibility and over-budget warnings, not filesystem enforcement.
+
+If a container was configured outside tx9 with unlimited CPU or RAM, use
+`tx9 upgrade <box>` plus the desired resource flags to move it to a finite
+limit. Docker's live update API cannot safely roll that transition back.
+
 Fresh and restored boxes begin with the Hermes gateway durably disabled so
 setup and migration cannot create a second Discord writer; enabling it is
 always an explicit, confirmed step. Do not use `hermes gateway run` as the
@@ -109,6 +160,10 @@ before creating anything, stage-then-promote inside a throwaway container,
 re-arm the quiesce and gateway-disable markers, mint a fresh executor token,
 and rewire MCP automatically.
 
+Agent-side logs and native histories travel with a normal `.tx9` backup.
+Executor runtime logs remain on its deliberately non-portable scratch volume;
+use `tx9 logs export` when those should travel too.
+
 ## Local validation
 
 ```bash

box.env

@@ -4,12 +4,15 @@
 # Apple Silicon smolvm pulls the native arm64 variant (no emulation).
 BASE_IMAGE="ubuntu:24.04"
 
-# VM sizing. Values are validated before smolvm is called. STORAGE_GIB is optional;
-# the 64 GiB overlay leaves room for a large Hermes database and backup staging.
-BOX_CPUS="4"
-BOX_MEM_MIB="8192"
-BOX_OVERLAY_GIB="64"
-BOX_STORAGE_GIB=""
+# Runtime sizing is per box and managed by `tx9 resources`, not baked into
+# the image. Existing defaults remain agent=4 CPU/8 GiB and executor=2 CPU/
+# 2 GiB. Docker local volumes are host-backed and unbounded; tx9 can attach
+# advisory storage budgets without pretending they are filesystem quotas.
+
+# Durable process logs rotate at 20 MiB per file, keeping five generations.
+# Native Codex, Claude, and Hermes histories retain their own upstream layout.
+TX9_LOG_MAX_BYTES="20971520"
+TX9_LOG_MAX_FILES="5"
 
 # Node.js is managed by Vite+ (https://viteplus.dev), defaulted to the
 # current LTS at provision time (`vp env default lts`). claude and codex use

docker/entrypoint.sh

@@ -1,9 +1,8 @@
 #!/usr/bin/env bash
 # Container PID-1 workload (run under docker --init so signals behave).
 # Reuses guest/hb-workload verbatim: it reconciles Executor, the Hermes
-# gateway, and the 0.0.0.0 socat bridges every 20s. The outer loop is the
-# supervisor for the loop process itself; Docker's restart policy is the
-# supervisor for the container.
+# gateway, and the 0.0.0.0 socat bridges every 20s. tx9-logs supervises the
+# loop process itself; Docker's restart policy supervises the container.
 set -uo pipefail
 
 trap 'exit 143' TERM
@@ -40,11 +39,21 @@ configure_mount_groups || exit 1
 # healthy, so the restart policy (not a silent loop) handles recovery.
 /opt/hermes-box/bin/hb init || exit 1
 
+# Runtime log limits are image defaults in the managed box environment. They
+# are passed explicitly because runuser starts tx9-logs without a login shell.
+# shellcheck disable=SC1091
+[ ! -r /etc/hermes-box.env ] || . /etc/hermes-box.env
+
 EXEC_BRIDGE_PORT="${BOX_EXECUTOR_BRIDGE_PORT:-14788}"
 API_BRIDGE_PORT="${BOX_HERMES_BRIDGE_PORT:-18642}"
 
-while true; do
-  runuser -u agent -- env HOME=/data/home/agent \
-    /opt/hermes-box/bin/hb-workload "$EXEC_BRIDGE_PORT" "$API_BRIDGE_PORT" || true
-  sleep 2
-done
+# tx9-logs owns the existing restart loop so it can forward container signals
+# to the active workload process group while mirroring redacted output to
+# Docker and persisting workload.log + normalized agent.jsonl on the volume.
+exec runuser -u agent -- env HOME=/data/home/agent \
+  TX9_BOX_NAME="${TX9_BOX_NAME:-}" \
+  TX9_LOG_MAX_BYTES="${TX9_LOG_MAX_BYTES:-20971520}" \
+  TX9_LOG_MAX_FILES="${TX9_LOG_MAX_FILES:-5}" \
+  /opt/hermes-box/bin/tx9-logs capture \
+    --source agent --log-dir /data/logs --restart-delay 2 -- \
+    /opt/hermes-box/bin/hb-workload "$EXEC_BRIDGE_PORT" "$API_BRIDGE_PORT"

docker/executor-entrypoint.sh

@@ -14,8 +14,13 @@ set -euo pipefail
 mkdir -p /data/home/agent /data/logs
 chown agent:agent /data/home/agent /data/logs
 
+# tx9-logs remains in the foreground between runuser and Executor: it mirrors
+# stdout/stderr to Docker, persists redacted executor.log + executor.jsonl,
+# forwards signals to Executor's process group, and exits with its status.
 # shellcheck disable=SC2016 # $EXECUTOR_MCP_TOKEN expands in the inner shell
-exec runuser -u agent -- env HOME=/data/home/agent \
+exec runuser -u agent -- env HOME=/data/home/agent TX9_BOX_NAME="${TX9_BOX_NAME:-}" \
   bash --noprofile --norc -c '. /etc/profile.d/hermes-box.sh
-    exec executor daemon run --foreground --hostname 0.0.0.0 --port 4788 \
-      --auth-token "$EXECUTOR_MCP_TOKEN"'
+    export TX9_LOG_MAX_BYTES TX9_LOG_MAX_FILES
+    exec /opt/hermes-box/bin/tx9-logs capture --source executor --log-dir /data/logs -- \
+      executor daemon run --foreground --hostname 0.0.0.0 --port 4788 \
+        --auth-token "$EXECUTOR_MCP_TOKEN"'

docs/docker-architecture.md

@@ -79,6 +79,18 @@ per box into `.boxd/<name>.env` (mode 0600) and injects into both containers.
 with that token (`EXECUTOR_HOST` + `BOXD_EXECUTOR_TOKEN`); the same token
 authenticates you to the dashboard/MCP from the LAN or Tailscale.
 
+Both volumes also retain observability data without crossing the isolation
+boundary. Agent supervisor/Hermes output and native Codex/Claude/Hermes
+histories stay on `agent-data`; Executor runtime output stays on `exec-data`.
+`tx9 logs` mounts both volumes read-only into a short-lived helper solely for
+query/export. The agent container never receives the Executor volume.
+
+Container resource defaults are agent 4 CPU/8 GiB and Executor 2 CPU/2 GiB.
+They are per-box, live-updatable limits rather than image settings. Docker
+local volumes have host-wide elastic capacity on the supported default
+backend, so tx9 reports real usage plus `unlimited`, with optional advisory
+budgets for planning and warnings.
+
 What the agent container can do to Executor: exactly what any LAN client can
 do — talk HTTP to the token-gated endpoint. Nothing else. Cross-box: each
 compose project gets its own network, so `alpha`'s agent cannot even resolve

docs/tx9-cli-design.md

@@ -1,10 +1,10 @@
 # tx9 CLI — design
 
-Status: agreed 2026-07-04 (12 decisions below, settled in conversation with
-Davis). Nothing here is implemented yet; this document is the spec the Go
-implementation is built from. The bash prototypes that informed it (`./box`
-on smolvm, `./docker/boxd` on compose) are deleted; recover them from git
-history (`9350212` and earlier) when porting logic.
+Status: implemented and evolving. The initial 12 decisions were settled on
+2026-07-04; this document now records the shipped Go CLI plus later command
+surface additions. The deleted bash prototypes that informed it (`./box` on
+smolvm, `./docker/boxd` on compose) remain available in git history
+(`9350212` and earlier).
 
 ## What tx9 is
 
@@ -38,6 +38,11 @@ tx9 create
 
 ## Command surface
 
+Running `tx9` with no arguments prints an ASCII overview of configured boxes
+(state, image version, dashboard URL, container CPU/RAM, and volume
+usage/budgets), followed by the command list. It exits successfully when the
+Docker overview can be read. `tx9 help` remains Docker-independent.
+
 | Command | Behavior |
 |---|---|
 | `tx9 create [name]` (alias `new`) | Generate name if absent. Build `tx9-box:<version>` if missing (with real progress UX). Create network + volumes + both containers, mint token, wire MCP, run doctor. Print getting-started checklist. |
@@ -47,6 +52,8 @@ tx9 create
 | `tx9 backup <box>` (aliases `export`, `save`) | Flags: `--path` (default `~/Downloads`), `--password`/env/prompt, `--no-encrypt`. Quiesce → archive agent /data → validate → (encrypt) → verify → `<box>-<timestamp>.tx9`. |
 | `tx9 import <file.tx9>` (aliases `load`, `restore`) | Flags: `--name`, `--password`/env/prompt. Validate before creating anything; restore staged; arrive quiesced + gateway-disabled + fresh token; fail on name collision. |
 | `tx9 mount <add\|list\|remove> ...` | Persist host-directory bind mounts for an agent and recreate only its disposable container. Targets must be below `/mnt`, outside the portable `/data` volume. `add` supports `--read-only` and `--require-mountpoint`. |
+| `tx9 logs <box>` | Query durable agent, Executor, Hermes, Codex, and Claude events. Filters include source, age, text, count, and normalized JSONL. `tx9 logs export <box>` creates a mode-0600 portable log bundle from both isolated volumes. |
+| `tx9 resources <box>` | Show actual container CPU/RAM limits and volume usage versus advisory budgets. `resources set` updates limits live and persists them; `resources reset` restores 4 CPU/8 GiB (agent) and 2 CPU/2 GiB (Executor). |
 | `tx9 gateway <status\|enable\|disable> <box>` | Inspect or control the container-supervised Hermes gateway. Enable requires `--confirm-single-writer`. |
 | `tx9 open <box>` | Print (or open) the authenticated dashboard URL (`?_token=`). |
 | `tx9 doctor <box>` | In-box `hb doctor` + host-side published-port probe. |
@@ -58,6 +65,10 @@ tx9 create
 `--executor-web-base-url`, `--executor-publish`, and `--executor-dns` options.
 Their resolved values are persisted per box and reused on future upgrades.
 `--clear-executor-config` returns a box to the default dynamic HTTP exposure.
+The same commands also accept `--agent-cpus`, `--agent-memory`,
+`--executor-cpus`, `--executor-memory`, `--agent-volume-budget`, and
+`--executor-volume-budget`; omitted values keep the defaults or the box's
+current allocation, as appropriate.
 
 ## Fixed contracts (carried from the verified bash draft)
 
@@ -86,6 +97,17 @@ the port:
   `--require-mountpoint` refuses to bind an empty underlying directory when a
   host share is offline. Required non-root source GIDs are added to the agent
   container automatically.
+- **Resources**: container limits are per-box desired state, with current
+  defaults of 4 CPU/8 GiB for the agent and 2 CPU/2 GiB for Executor. Live
+  changes survive upgrades and agent recreation. Local named volumes remain
+  unbounded host-backed storage; recorded volume budgets are advisory and
+  never labeled as enforced quotas.
+- **Observability**: tx9-owned process output is redacted, rotated, and stored
+  on the producing component's own volume. Host-side queries merge these
+  streams with tolerant readers for the native Codex/Claude JSONL and Hermes
+  SQLite histories. Executor data is never mounted into the agent container.
+  This provides complete tx9-owned runtime output, not an independent audit
+  trail for operations that Executor does not emit to any durable sink.
 - **Gateway single-writer**: restored boxes never auto-enable the Hermes
   gateway. `tx9 gateway enable <box> --confirm-single-writer` delegates to
   `hb gateway-enable` and stays the only host-side path.