Quaternion deviation logging, MAVLink query tooling, IC-capture fix, venv centralization

.gitignore

@@ -18,8 +18,12 @@ env/
 simulation/logs/
 *.log
 
-# One-off / diagnostic scripts (kept locally for reproducibility, not checked in by default)
-tests/oneoff/
+# One-off / diagnostic scripts (kept locally for reproducibility, not checked in
+# by default). Matched at any depth -- e.g. simulation/tests/oneoff/ is a stray
+# legacy path that predates tests/oneoff/ moving to the repo-root tests/ package
+# (see AGENTS.md repo layout); anchoring this to root only would silently leave
+# that directory's scratch scripts untracked-but-visible to `git status`.
+**/tests/oneoff/
 
 # Generated flight report plots
 *.png

AGENTS.md

@@ -118,12 +118,23 @@ Use the primary doc for each topic. Other docs should link, not restate.
 | Hardware assembly and components | `design/hardware.md` | `design/components.md`, `design/dshot.md`, `design/flap_sensor_bench.md` |
 | Testing taxonomy and Lua/Python test conventions | `design/testing.md` | `pyproject.toml` (`[tool.pytest.ini_options]`) |
 | Milestones and decisions history | `design/history.md` | this file (summary only) |
+| MAVLink `*.mavlink.jsonl` log inspection (calibrate `run`, SITL stack tests) | `analysis/mavlink_jsonl_query.md` | `design/calibration.md` |
 
 Parameter-reference ownership note:
 - Canonical place for ArduPilot parameter defaults and inline explanations is `tests/sitl/copter-heli.parm`.
 - Canonical place for RAWES_* parameter defaults and inline explanations is `tests/sitl/rawes_common_defaults.parm`.
 - If a parameter explanation changes, update the owning `.parm` file first; other docs should link to it instead of duplicating bitmasks/tables.
 
+## MAVLink Log Diagnosis (Agent Critical)
+
+For ANY problematic run that produced a `*.mavlink.jsonl` log (calibrate `run`,
+SITL stack tests), use `analysis/mavlink_jsonl_query.py` as the first-line
+diagnostic tool -- before writing one-off jsonl-parsing code or manually
+grepping the raw file. See `analysis/mavlink_jsonl_query.md` for the full
+interface (subcommands, filters, gotchas); do not duplicate its contents
+here. If a diagnosis need doesn't fit an existing subcommand, prefer
+extending the script (new subcommand/filter) over a standalone script.
+
 ## Core Invariants (summary)
 
 - Frames: simulation physics runs NED world + FRD body.

analysis/mavlink_jsonl_query.md

@@ -0,0 +1,109 @@
+# `mavlink_jsonl_query.py` -- MAVLink JSONL log query tool
+
+Canonical usage doc for `analysis/mavlink_jsonl_query.py`, the standard,
+first-line tool for diagnosing any problematic run that produced a
+`*.mavlink.jsonl` log (calibrate `run`, SITL stack tests). Prefer this tool
+over writing one-off jsonl-parsing code or manually grepping the raw file. If
+a diagnosis need doesn't fit an existing subcommand, prefer extending this
+script (new subcommand/filter) over a standalone script -- keep this doc in
+sync with any command/flag changes.
+
+Referenced from `AGENTS.md` (Documentation Ownership table + "MAVLink Log
+Diagnosis" section) and `design/calibration.md`.
+
+## Log format
+
+Logs are written by `MavlinkLogWriter` (`groundstation/mavlink_log.py`) from
+both `groundstation/gcs.py` (`RawesGCS.start_mavlog`) and the bench
+calibration tool (`calibrate/run.py`). Every line is one JSON object:
+
+```json
+{"_t_wall": <float>, "_dir": "rx"|"tx", "mavpackettype": "<TYPE>", ...fields...}
+```
+
+`_t_wall` is wall-clock seconds; `_dir` is `"rx"` (from the FC) or `"tx"` (to
+the FC). The tool adds a derived `t_rel` field (seconds since the first
+message in the log) to every message before filtering/printing.
+
+## Subcommands
+
+```
+mavlink_jsonl_query.py types      <log.jsonl>
+mavlink_jsonl_query.py show       <log.jsonl> [filters] [--fields a,b,c] [--json] [--limit N]
+mavlink_jsonl_query.py count      <log.jsonl> [filters] [--by FIELD]
+mavlink_jsonl_query.py stats      <log.jsonl> --type T --field F [filters]
+mavlink_jsonl_query.py armed      <log.jsonl>
+mavlink_jsonl_query.py statustext <log.jsonl> [--since S] [--until S] [--min-severity N]
+mavlink_jsonl_query.py nvf        <log.jsonl> [--name RAWES_ARM] [--dir rx|tx]
+mavlink_jsonl_query.py param      <log.jsonl> [--id RAWES_MODE]
+```
+
+Run `--help` on any subcommand for the exact flags; this doc gives the intent
+and gotchas for each.
+
+- **`types`** -- list every `mavpackettype` seen, split by direction, with
+  count and first/last `t_rel`. Always the right first command on an unknown
+  log: confirms whether a message type is present at all before chasing a
+  decode bug. A message type with count 0 (i.e. absent from this list) means
+  it was never requested/streamed from the FC -- not a decode/registration
+  bug in `groundstation/gcs.py`.
+- **`show`** -- generic filtered dump of raw message fields, one line per
+  message. Use `--fields` to restrict columns, `--json` to emit raw JSON
+  lines (for piping into `jq`/Python), `--limit` to cap output. This is the
+  fallback for any message type without a dedicated subcommand; it does NOT
+  do any special-casing (e.g. no STATUSTEXT reassembly -- use the
+  `statustext` subcommand for that).
+- **`count`** -- count matched messages, optionally grouped by a field
+  (`--by mavpackettype`, `--by mode`, etc).
+- **`stats`** -- min/max/mean/median/stdev for one numeric field, e.g.
+  `--type ATTITUDE --field yaw`.
+- **`armed`** -- timeline of HEARTBEAT armed-state/mode transitions (rx only,
+  excludes GCS self-heartbeats). Good for confirming arm/disarm timing and
+  flight-mode transitions without wading through every HEARTBEAT.
+- **`statustext`** -- dump `gcs:send_text()`/STATUSTEXT messages in order,
+  **reassembled** from ArduPilot's wire-level chunking. See "STATUSTEXT
+  reassembly" below -- this is the one subcommand where NOT using the tool
+  (i.e. grepping the raw jsonl) actively produces misleading output.
+- **`nvf`** -- dump the `NAMED_VALUE_FLOAT` stream, i.e. the ground<->Lua
+  `RAWES_*` interface. Filter with `--name RAWES_ARM` to follow one channel,
+  `--dir tx` to see only what the ground sent.
+- **`param`** -- dump `PARAM_SET`/`PARAM_VALUE` events, filter with `--id`.
+
+## Common filters (`show`/`count`/`stats`)
+
+- `-t/--type TYPE` (repeatable; OR) -- e.g. `-t ATTITUDE -t AHRS2`
+- `--dir {rx,tx}`
+- `--since SEC` / `--until SEC` -- `t_rel` bounds
+- `--eq FIELD=VALUE` (repeatable; AND) -- e.g. `--eq mavpackettype=STATUSTEXT`
+- `--contains TEXT` -- substring match on the raw JSON line (case-insensitive)
+
+## STATUSTEXT reassembly (agent-critical)
+
+ArduPilot's `gcs:send_text()`/STATUSTEXT wire format caps the `text` field at
+50 bytes. Any Lua `gcs:send_text()` call longer than that is split across
+multiple STATUSTEXT messages that share a common non-zero `id` field, ordered
+by `chunk_seq` (0, 1, 2, ...). `id == 0` is the ArduPilot sentinel for a
+standalone, already-short single-chunk message -- those must never be merged
+with each other even if several appear back-to-back.
+
+`mavlink_jsonl_query.py statustext` reassembles chunks sharing the same
+non-zero `id` into a single logical line before printing. Do not use `show
+-t STATUSTEXT` or grep the raw jsonl directly when reading Lua diagnostic
+text -- both will print each ~50-byte fragment as its own line, sometimes
+splitting a message mid-word (e.g. `"RAWES YIC capture: r=89.9 p=20.0 y=-175.3"`
+can arrive on the wire as `"RAWES YIC capture: r=89.9 p=20.0 y=-17"` +
+`"5.3"`), which is hard to read for both humans and AI and can look like log
+corruption when it's just unreassembled chunking.
+
+## Example: diagnosing a "target never updates" bug
+
+```
+mavlink_jsonl_query.py types      simulation/logs/calibrate/run_passive_....mavlink.jsonl
+mavlink_jsonl_query.py statustext simulation/logs/calibrate/run_passive_....mavlink.jsonl
+mavlink_jsonl_query.py show -t ATTITUDE_TARGET --fields q simulation/logs/calibrate/run_passive_....mavlink.jsonl
+```
+
+`types` confirms the message is actually present (vs. a missing stream
+request); `statustext` surfaces Lua's own diagnostic text (e.g. `RAWES YIC
+capture: ...`, `RAWES guided cmd: ...`) to compare what Lua *thinks* it
+captured/sent against what the FC is actually reporting.

analysis/mavlink_jsonl_query.py

@@ -8,10 +8,13 @@
 
     {"_t_wall": <float>, "_dir": "rx"|"tx", "mavpackettype": "<TYPE>", ...fields...}
 
-This is the standard tool for ad-hoc inspection of these logs -- prefer
-extending this script (new subcommand or filter) over writing one-off jsonl
-parsing code. See mavlink_jsonl_query.md in this directory for usage
-docs/examples; keep that file in sync with any command/flag changes.
+This is the standard, first-line tool for diagnosing any problematic run that
+has a `*.mavlink.jsonl` log (calibrate `run`, SITL stack tests) -- prefer it
+over writing one-off jsonl parsing code or manually grepping the raw file, and
+prefer extending this script (new subcommand or filter) over either of those.
+See mavlink_jsonl_query.md in this directory for the canonical usage doc
+(subcommand intent, gotchas, examples); keep that file in sync with any
+command/flag changes.
 
 Usage summary (see `--help` on each subcommand for full options):
     mavlink_jsonl_query.py types      <log.jsonl>
@@ -229,19 +232,68 @@ def cmd_armed(args: argparse.Namespace) -> None:
             last_mode = mode
 
 
+def _reassemble_statustext(rows: Iterable[dict]) -> Iterator[dict]:
+    """Reassemble multi-chunk STATUSTEXT messages into single logical lines.
+
+    ArduPilot's gcs:send_text()/STATUSTEXT wire format caps `text` at 50 bytes;
+    longer messages are split across multiple STATUSTEXT messages that share a
+    common non-zero `id`, ordered by `chunk_seq` (0, 1, 2, ...). `id == 0` is
+    the ArduPilot sentinel for a standalone, already-short single-chunk
+    message -- those must NEVER be merged with each other even if several
+    appear back-to-back. Without this reassembly, `show`/raw dumps print each
+    ~50-char fragment as its own line/timestamp, which is hard for both humans
+    and AI to read (a single log line like "RAWES YIC capture: r=89.9 p=20.0
+    y=-175.3" can be split mid-word across two lines).
+    """
+    pending_id: "int | None" = None
+    pending_chunks: dict[int, str] = {}
+    pending_meta: "dict | None" = None
+
+    def _flush() -> "dict | None":
+        if pending_meta is None:
+            return None
+        out = dict(pending_meta)
+        out["text"] = "".join(pending_chunks[k] for k in sorted(pending_chunks))
+        return out
+
+    for m in rows:
+        mid = m.get("id", 0)
+        text = m.get("text", "")
+        if isinstance(text, (bytes, bytearray)):
+            text = text.decode("utf-8", errors="replace")
+        text = text.split("\x00", 1)[0]
+        if mid == 0:
+            flushed = _flush()
+            if flushed is not None:
+                yield flushed
+            pending_id, pending_chunks, pending_meta = None, {}, None
+            out = dict(m)
+            out["text"] = text
+            yield out
+            continue
+        if mid != pending_id:
+            flushed = _flush()
+            if flushed is not None:
+                yield flushed
+            pending_id = mid
+            pending_chunks = {}
+            pending_meta = dict(m)
+        pending_chunks[m.get("chunk_seq", 0)] = text
+    flushed = _flush()
+    if flushed is not None:
+        yield flushed
+
+
 def cmd_statustext(args: argparse.Namespace) -> None:
     msgs = load(args.log)
-    for m in apply_filters(msgs, types=["STATUSTEXT"], direction="rx",
-                            since=args.since, until=args.until):
+    rows = apply_filters(msgs, types=["STATUSTEXT"], direction="rx",
+                          since=args.since, until=args.until)
+    for m in _reassemble_statustext(rows):
         sev = m.get("severity", -1)
         if args.min_severity is not None and sev > args.min_severity:
             # NOTE: lower severity number == more severe (MAVLink convention)
             continue
-        text = m.get("text", "")
-        if isinstance(text, (bytes, bytearray)):
-            text = text.decode("utf-8", errors="replace")
-        text = text.split("\x00", 1)[0]
-        print(f"[{m['t_rel']:>9.3f}s] {MAV_SEVERITY.get(sev, sev):<9} {text}")
+        print(f"[{m['t_rel']:>9.3f}s] {MAV_SEVERITY.get(sev, sev):<9} {m['text']}")
 
 
 def cmd_nvf(args: argparse.Namespace) -> None:

calibrate.cmd

@@ -2,43 +2,9 @@
 setlocal
 
 set "ROOT=%~dp0"
-set "VENV=%ROOT%.venv"
-set "PYTHON=%VENV%\Scripts\python.exe"
-set "PIP=%VENV%\Scripts\pip.exe"
-set "REQ=%ROOT%simulation\requirements.txt"
-set "STAMP=%VENV%\.req_stamp"
+set "PYTHON=%ROOT%.venv\Scripts\python.exe"
 
-if not exist "%VENV%\Scripts\activate.bat" (
-    echo Creating venv at %VENV% ...
-    python -m venv "%VENV%"
-    if errorlevel 1 (
-        echo ERROR: Failed to create venv. Is Python 3 on PATH?
-        exit /b 1
-    )
-)
-
-if not exist "%PYTHON%" (
-    echo ERROR: venv python not found at %PYTHON%
-    exit /b 1
-)
-
-:: Only reinstall if requirements.txt changed since last install
-for /f "delims=" %%H in ('certutil -hashfile "%REQ%" MD5 ^| findstr /v "hash"') do set REQ_HASH=%%H
-set REQ_HASH=%REQ_HASH: =%
-
-set STAMP_HASH=
-if exist "%STAMP%" (
-    set /p STAMP_HASH=<"%STAMP%"
-)
-
-if not "%REQ_HASH%"=="%STAMP_HASH%" (
-    echo Installing/verifying requirements ...
-    "%PIP%" install -q -r "%REQ%"
-    if errorlevel 1 (
-        echo ERROR: pip install failed.
-        exit /b 1
-    )
-    echo %REQ_HASH%>"%STAMP%"
-)
+call "%ROOT%setup.cmd"
+if errorlevel 1 exit /b 1
 
 "%PYTHON%" -m calibrate %*

calibrate/constants.py

@@ -14,6 +14,7 @@
 # Re-exported so submodules can do `from .constants import RawesGCS` etc.
 from groundstation.gcs         import (
     Attitude,
+    AttitudeQuaternion,
     EscTelemetry,
     PidTuning,
     BatteryStatus,