Last verified 2026-08-02
fleetctl keeps three observability streams that look similar on the surface but serve different consumers and carry different security postures. Conflating them — which is what happened in firestick_manager, whose CLI never called logging.basicConfig so every diagnostic line was silently discarded — produces both an unreadable audit trail and a real risk of credential leaks in debug output.
| Stream | What it is | Consumer | Retention |
|---|---|---|---|
| Diagnostic logging | LOGGER calls, %s-formatted, to stderr via configure_logging() |
Developer debugging a run, -v/-vv |
Not persisted by fleetctl itself — a caller (systemd, HA, a container) owns rotation if it wants any |
| Operation timeline | Human-readable progress — one line per major step, held on the Operation record |
CLI echo, fleetctl audit tail-adjacent polling via get_operation |
In-memory, bounded to 500 operations, oldest finished ones evicted first |
| Audit trail | Append-only, structured, hash-chained JSONL — one record per auditable effect | Anyone reviewing what happened, when, and who did it | Files accumulate under audit_dir; no automatic expiry today |
flowchart LR
subgraph src["Sources"]
ST["Step bodies"]
TR["AuditingTransport — every exec/put/get"]
EN["OperationRegistry — timeline + status"]
end
ST --> D & O
TR --> D & A
EN --> O
D["Diagnostic<br/>LOGGER calls, stderr"]
O["Timeline<br/>Operation.logs, in-memory"]
A["Audit<br/>append-only, hash-chained"]
D --> DC["stderr, annotated with run/step/op via CorrelationFilter"]
O --> OC["CLI echo / get_operation() polling"]
A --> AC["audit/YYYY-MM-DD.jsonl"]
Whether an effect is audited is keyed off the same classification steps already declare for policy purposes (see safety.md) — Effect.is_auditable is true for mutating and destructive, false for read. A read call still logs diagnostically; it just never reaches the audit trail. A fleet-wide maintenance run produces hundreds of read probes that never touch the durable stream, and the handful of pm disable-user calls that do change something end up as individual audit records — nothing is lost either way, and the durable stream stays small enough to actually review.
# config/fleet.yml
observability:
audit_dir: audit # relative to --home, default ~/.fleetctlThat's the entire config surface today — one path. Routing, redaction, and hash-chaining are not configurable; they're structural.
Every side effect on every device — regardless of which pack issued it — goes through Transport.exec / .put / .get. Wrapping the transport in AuditingTransport captures all of them automatically, for every step ever written, including a future third-party pack that doesn't know auditing exists.
classDiagram
class Transport {
<<Protocol>>
+exec(cmd, effect, timeout_s) str
+put(local, remote, effect) int
+get(remote, local) int
}
class AdbTransport
class AuditingTransport {
-inner: Transport
-audit: ChainedAuditWriter
+exec(cmd, effect) str
}
Transport <|.. AdbTransport
Transport <|.. AuditingTransport
AuditingTransport o-- Transport : decorates
AuditingTransport is constructed once, by the composition root (cli/bootstrap.py::Container.transport_for) — a step receives it already wrapped and cannot opt out, because it never constructs its own transport. This is what makes auditing a property of the wiring rather than an obligation every pack author has to remember.
flowchart TB
RUN["run_id — one workflow invocation"] --> STP["step_id — one step"]
STP --> OP["op_id — one operation"]
RUN -.->|"correlate() ContextVar + CorrelationFilter"| LOGS["every LOGGER line carries run/step/op/actor"]
core/observability/correlation.py::correlate() binds run_id/step_id/op_id/actor on a ContextVar for the duration of a block; CorrelationFilter (a logging.Filter) injects the current values into every log record. This closes a concrete gap in firestick_manager: its service runs several jobs concurrently on a thread pool, and interleaved log output from simultaneous deploys is unattributable after the fact. fleetctl's background Dispatcher binds correlation inside each worker thread specifically because context variables don't cross a thread boundary on their own — losing that would have reintroduced the same gap.
@dataclass(frozen=True, slots=True)
class AuditEvent:
"""One recorded effect. Never updated after it is written."""
kind: AuditKind # exec | put | get | plan | config | decision | auth
action: str # "pm disable-user" | "artifact.put"
effect: Effect = Effect.MUTATING
outcome: Outcome = Outcome.OK # ok | failed | skipped | denied
target: str = "" # device address or id
detail: Mapping[str, Any] = field(default_factory=dict) # redacted before write
error: str | None = None
duration_ms: int = 0
ts: str = ... # ISO-8601 UTC
run_id: str = "-"
step_id: str = "-"
op_id: str = "-"
actor: str = "-"
seq: int = 0
prev_hash: str = "" # hash chain over the preceding record
hash: str = ""Written as JSONL, one file per day, appended and never rewritten. prev_hash chains each record to the one before it, so fleetctl audit verify can detect truncation or a mid-file edit cheaply — the difference between "a log" and "a record you can trust when running an incident down." The chain resumes correctly across process restarts: a fresh CLI invocation continues the existing day's chain rather than re-anchoring at genesis.
Four event kinds beyond raw exec/put/get are worth naming:
config— a resolved configuration decision, so "why did this stick get that setting?" is answerable without reading Python.decision— a policy verdict: an approval requirement, a denial, a blast-radius cap hit. Every branch that denies is itself an audited event.auth— recorded around ADB key use where relevant.plan— a workflow plan, recorded whether or not a run follows, so intent is on the record even when nothing executes (seesafety.md's plan-then-run flow).
core/observability/redact.py::Redactor strips a default set of sensitive key names (password, secret, token, api_key, auth, credential, private_key, and close variants) out of anything headed for the audit trail or a log line, applied inside AuditingTransport so no step or transport implementation can bypass it. Separately, config values wrapped in Secret (core/config/secrets.py) render as a fixed mask under str()/repr() — a caller must call .reveal() deliberately to get the underlying value, which is the one thing standing between a resolved !ref and it leaking into a log line or an accidental print().
Every operation runs inside a staging directory under --home/staging. On a non-cancelled failure, container.failures_root (--home/forensics) preserves that workspace rather than letting it get torn down with the rest — before-the-fact evidence for whatever failed, without needing to reproduce it. Retention is capped to the most recent entries.
- How the audit trail feeds the policy layer's approval and denial decisions:
safety.md - Every field in the
observability:config block:configuration.md audit tail/audit verifycommands:cli-reference.md
|
fleetctl Observability |
|
|