Add box observability and resource controls

[bmdavis419]

Summary

• persist rotated structured runtime logs and normalize native Codex, Claude, and Hermes histories for filtered query/export
• expose and update per-box agent/Executor CPU and memory limits plus actual volume usage and advisory budgets while preserving existing defaults
• show a sorted ASCII box/resource overview when tx9 runs without arguments

Operational boundaries

• Docker local named volumes do not provide a portable hard quota; volume budgets are explicit advisory thresholds, while usage is measured from Docker
• logging captures tx9-owned process output and persisted upstream histories; it cannot reconstruct events an upstream tool never emits

Validation

• make check
• git diff --check
• independent frozen-diff logging/security and gateway concurrency reviews
• local Docker resource update API proof for memory and swap behavior

---

Note

Add box observability with tx9 logs, tx9 resources, and structured log capture via tx9-logs

• Adds tx9 logs [export] <box> to query or export structured logs from agent/executor volumes via a transient container running the new guest/tx9-logs helper, with filtering, redaction, and JSON output options.
• Adds tx9 resources show/set/reset <box> to inspect and live-update per-box CPU/memory limits and advisory volume budgets, with transactional rollback on failure.
• Introduces guest/tx9-logs, a Python3 logging daemon that wraps subprocesses, redacts sensitive tokens, rotates logs, and persists structured .jsonl logs; both agent and executor entrypoints now use it instead of shell redirect loops.
• Running tx9 with no arguments now shows an ASCII overview panel with container resource usage, volume usage, and dashboard URLs instead of printing usage and exiting 1.
• tx9 create, tx9 import, and tx9 upgrade now accept resource flags (CPU/memory/budget) that are validated, persisted, and applied to containers at creation or recreation time.
• docker.Client gains VolumeUsage and ContainerUpdateResources; WriteBoxEnv now performs an atomic temp-file-and-rename write.
• Risk: agent and executor container entrypoints are replaced with tx9-logs-supervised exec, changing signal handling and restart behavior for existing boxes.

^{Macroscope summarized b94eb01.}

Greptile Summary

This PR adds box observability and resource controls. The main changes are:

• Durable structured logs for agent, Executor, Hermes, Codex, and Claude activity.
• tx9 logs query and export commands.
• tx9 resources show, set, and reset commands.
• Per-box CPU, memory, and advisory volume budget settings.
• A no-argument ASCII overview for configured boxes.
• Atomic env-file writes for persisted box settings.

Confidence Score: 5/5

This looks safe to merge.

No blocking issues found in the changed code.

T-Rex Logs

What T-Rex did

• T-Rex produced a proof for the posted P1 finding.
• T-Rex ran the requested verification, but its local artifact references were not uploaded.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
internal/cli/dispatch.go	Adds the no-argument overview path and keeps the command list fallback available when the overview cannot load.
internal/cli/overview_runtime.go	Builds the Docker-backed overview with bounded context use and degraded output for unavailable runtime details.
internal/cli/cmd_resources.go	Adds resource inspection and update flows with live Docker updates, persistence, and rollback handling.
internal/box/resources.go	Adds persisted resource defaults, parsing, formatting, validation, and reset behavior.
internal/state/state.go	Changes box env writes to use a same-directory temporary file followed by rename.

Comments Outside Diff (6)

1. General comment

   Log regression script fails because jq -e select... returns the status of the final non-matching JSONL record

   • Bug
     • ./tests/regressions-logs.sh fails at its first JSONL assertion with exit code 4. The captured executor.jsonl contains stdout output records that match the filter, but the file also ends with non-matching records such as stderr/process_exit. With jq -e 'select(...)' file, jq's exit status reflects the last output/input evaluation behavior and is 4 when the final result is false/null/no output, even if earlier records matched and were printed. This makes the regression script fail on valid structured output.
   • Cause
     • The test uses jq -e 'select(.source == "executor" and .stream == "stdout" and .type == "output")' executor.jsonl >/dev/null as an existence assertion over a stream of JSONL records. Because later non-matching records are evaluated after matching records, the command exits 4.
   • Fix
     • Change the existence assertion to aggregate over all JSONL records, for example jq -e 'any(inputs; .source == "executor" and .stream == "stdout" and .type == "output")' < executor.jsonl or jq -e 'select(...)' executor.jsonl | grep -q . with pipefail handled appropriately. Apply the same pattern to similar JSONL existence checks if present.

   _{Ran code and verified through T-Rex}

2. General comment

   Logging regression script fails on first executor JSONL output assertion

   • Bug
     • ./tests/regressions-logs.sh fails almost immediately after running guest/tx9-logs capture --source executor ...; the traced run exits at jq -e 'select(.source == "executor" and .stream == "stdout" and .type == "output")' "$executor_root/logs/executor.jsonl" >/dev/null with EXIT_CODE: 4. This means the PR's explicit log regression validation is not passing in the sandbox, despite syntax/static checks and changed Go package tests passing.
   • Cause
     • The generated executor.jsonl from the first regression fixture does not satisfy the regression script's required stdout output-record selector, causing jq -e to return no truthy result under set -e. The trace shows failure before the later redaction, rotation, query, and export checks can run.
   • Fix
     • Inspect guest/tx9-logs capture output for the first fixture in tests/regressions-logs.sh and restore emission of an executor stdout record with source: "executor", stream: "stdout", and type: "output" for captured stdout, or update the regression only if the schema intentionally changed and all consumers were migrated.

   _{Ran code and verified through T-Rex}

3. General comment

   Log regression suite fails at first structured executor JSONL assertion

   • Bug
     • make test fails in the changed log regression path. A diagnostic bash -x ./tests/regressions-logs.sh rerun shows the script exits with status 4 immediately after invoking jq -e 'select(.source == "executor" and .stream == "stdout" and .type == "output")' "$executor_root/logs/executor.jsonl". The wrapped capture command itself returned the expected status 7 and the raw stdout/stderr/log files were created with expected permissions before the failure point, so the failure is localized to the structured JSONL assertion path rather than missing Docker or shellcheck.
   • Cause
     • The regression script's first structured log verification command returns jq exit status 4 in this environment. The available trace shows the failure occurs at the jq -e select(...) assertion in tests/regressions-logs.sh lines 29-30; no product-side diagnostic is emitted by the script before cleanup removes the temp directory.
   • Fix
     • Preserve or print the generated executor.jsonl on assertion failure and adjust the structured log generation or assertion so at least one executor stdout output event matching .source == "executor" and .stream == "stdout" and .type == "output" is present and accepted. Then rerun ./tests/regressions-logs.sh and make test.

   _{Ran code and verified through T-Rex}

4. General comment

   Logging regression script fails at jq selection despite valid event data

   • Bug
     • ./tests/regressions-logs.sh exits with status 4 during the first JSONL validation step: jq -e 'select(.source == "executor" and .stream == "stdout" and .type == "output")' executor.jsonl >/dev/null. A direct repro of the same capture path produced an executor.jsonl file containing a matching stdout output event, but jq -e can still return 4 when the final input record does not match the filter, because the script uses select(...) over all records rather than aggregating whether any record matched. This causes the regression suite to fail even though the expected event is present.
   • Cause
     • The test uses jq -e 'select(...)' file across multiple JSONL records. With -e, jq's exit status reflects the last output value; when later records are non-matching and produce no output, the command can exit 4 even if an earlier record matched.
   • Fix
     • Change the assertion to test for any matching record, for example jq -e 'any(inputs; .source == "executor" and .stream == "stdout" and .type == "output")' with appropriate JSONL input handling, or collect matches and assert length > 0.

   _{Ran code and verified through T-Rex}

5. General comment

   Changed logs regression script fails under jq 1.6 despite matching JSONL records

   • Bug
     • ./tests/regressions-logs.sh exits with status 4 at its first JSONL assertion: jq -e 'select(.source == "executor" and .stream == "stdout" and .type == "output")' "$executor_root/logs/executor.jsonl" >/dev/null. A direct repro shows guest/tx9-logs capture writes valid matching stdout/output records, but jq still returns 4 because the filter is evaluated across multiple JSONL objects and the final non-matching input produces no truthy output under jq -e.
   • Cause
     • The regression assertion relies on jq -e select(...) as an existence check over a multi-record JSONL file. With jq 1.6, the command's exit status can be 4 when the last processed input yields no output, even if earlier records matched and were printed.
   • Fix
     • Rewrite the assertion as an explicit any/existence check that returns one final boolean, for example jq -e 'any(inputs; .source == "executor" and .stream == "stdout" and .type == "output")' with appropriate handling for the first input, or use jq -e 'select(...)' file | grep -q . without relying on jq's multi-input -e exit status.

   _{Ran code and verified through T-Rex}

6. General comment

   Logging regression script fails despite matching JSONL events

   • Bug
     • ./tests/regressions-logs.sh exits 4 at its first JSONL assertion. The script runs jq -e 'select(.source == "executor" and .stream == "stdout" and .type == "output")' executor.jsonl >/dev/null; matching stdout events are present, but jq exits 4 because -e bases status on the last output value and the final JSONL record is a non-matching process_exit event.
   • Cause
     • The regression test uses jq -e select(...) over a multi-record JSONL stream without aggregating the predicate, so a later non-match can make jq return failure even after prior matches succeeded.
   • Fix
     • Change the assertion to aggregate matches, for example jq -e 'any(.source == "executor" and .stream == "stdout" and .type == "output")' executor.jsonl with slurp/input handling appropriate for JSONL, or jq -e '[select(...)] | length > 0' when slurping records.

   _{Ran code and verified through T-Rex}

_{Reviews (6): Last reviewed commit: "fix: validate capture restart delays" | Re-trigger Greptile}

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

tx9 platform workflows

Layer / File(s)	Summary
Capture, redaction, querying, and export guest/tx9-logs, tests/regressions-logs.sh	Adds bounded rotating logs, streaming credential redaction, multi-source querying, filtering, and portable gzip exports with regression coverage.
Runtime logging and gateway coordination docker/*entrypoint.sh, guest/hb*, provision/provision.sh, box.env, tests/regressions-hb-workload.sh	Runs workloads and executors through tx9-logs, centralizes bridge logging, provisions the helper, and coordinates gateway wrapper startup.
Resource model and Docker integration internal/box/resources.go, internal/box/create.go, internal/docker/*	Defines resource defaults, persistence, validation, inspection, volume usage, memory-swap handling, and container updates.
Resource lifecycle commands internal/cli/cmd_resources.go, internal/cli/cmd_create.go, internal/cli/cmd_import.go, internal/cli/cmd_upgrade.go	Adds resource flags and show, set, and reset flows, including transactional updates and rollback.
CLI logs and overview internal/cli/cmd_logs.go, internal/cli/overview*.go, internal/cli/dispatch.go	Adds logs query/export commands, no-argument ASCII overview rendering, Docker metrics, and command dispatch integration.
Documentation, state, and naming contracts README.md, docs/*.md, internal/state/*, internal/names/*	Documents portability, logs, resources, backups, and overview behavior; reserves logs and resources; and writes box environment files atomically.

Possibly related PRs

• davis7dotsh/tx9#2: Both modify the Makefile syntax checks and Hermes guest-script validation.
• davis7dotsh/tx9#19: Both modify CLI dispatch and reserved box-name validation.
• davis7dotsh/tx9#21: Both modify Hermes gateway startup coordination and related guest/hb wiring.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.35% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: adding box observability and resource controls.
Description check	✅ Passed	The description is detailed and directly related to the log, resource, overview, and update changes in the PR.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

guest/tx9-logs (1)

1541-1576: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

Avoid the always-on fingerprint map on the hot path

emitted_counts is populated for every direct-read row even when the read completes successfully, so this path uses O(unique events) memory before the fallback ever runs. Since hermes_database_events() reads both tables without a stable ORDER BY, the exact dedup state is doing real work here; if large Hermes DBs are expected, consider whether this recovery tradeoff is acceptable or whether the fallback can be redesigned around a cheaper identity scheme.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@guest/tx9-logs` around lines 1541 - 1576, The direct-read path in the
surrounding recovery flow always builds the O(unique events) emitted_counts
fingerprint map, even when no fallback is needed. Redesign the fallback
deduplication around a cheaper identity scheme or defer/limit state collection
so successful direct reads avoid retaining every event, while preserving correct
duplicate suppression when snapshot recovery follows a partial read.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@guest/tx9-logs`:
- Around line 1541-1576: The direct-read path in the surrounding recovery flow
always builds the O(unique events) emitted_counts fingerprint map, even when no
fallback is needed. Redesign the fallback deduplication around a cheaper
identity scheme or defer/limit state collection so successful direct reads avoid
retaining every event, while preserving correct duplicate suppression when
snapshot recovery follows a partial read.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c2143013-2511-4ace-be01-a9e80a82978e

📥 Commits

Reviewing files that changed from the base of the PR and between 9188321 and c9c50a5.

📒 Files selected for processing (2)

• guest/tx9-logs
• tests/regressions-logs.sh

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/regressions-logs.sh