Stabilize TX streaming: reconcile IT ring lap loss, fault dead IRQ path, classify replay misses

[mrmidi]

Symptom

A Duet duplex stream could degrade into a "transport-alive, audio-dead" zombie: FireBug showed both isoch channels running at one packet per cycle with zero silent cycles, TX SYT tracking RX SYT exactly — and every PCM sample in the outgoing DATA packets zero. The 2026-07-19 incident ran in this state for ~35 minutes with no fault raised.

Root cause (confirmed from the live MCP driver ring)

The measured chain, each link verified in code and in the retained ring (17.3-minute window, zero dropped records):

1. The IT descriptor ring is free-running (48 packets, circular; hardware never stops), and IsochTxDmaRing::ComputeDeltaConsumed() measured progress from the command pointer — which is only defined modulo 48. Any refill gap longer than 6 ms silently discarded whole laps: the wire re-transmitted the stale 48 slots (a DBC seam per lap, invisible in SYT because 48 ≡ 0 mod 16 cycles) while completionCursor under-advanced with no fault raised.
2. The IT interrupt path died ~10 minutes into the session (810,214 IRQs ≈ a healthy 1,333/s for the first ~10 min, then permanent silence). The Poll() watchdog (~68 ms cadence) carried the stream, losing ~10 ring laps per kick. Net: 71% of the session's wire packets were stale re-transmissions; completionCursor advanced at ~575 packets/s against the wire's 8,000/s.
3. The TX producer paces itself from completionCursor, so producer time dilated to ~7% of bus time. The RX replay reader consumed ~575 entries/s against RX's healthy 8,000/s and fell out of its 512-entry history window every few tens of milliseconds ([TxReplay] fail=overwritten, d=-547…-920, epoch unchanged — no RX reset involved).
4. Every replay miss reset the reader and re-armed frame-cursor alignment. The next DATA read re-projected the cursor ~10k frames ahead of the host write frontier, so AmdtpPayloadWriter classified essentially every host frame outside and the pre-zeroed DATA payloads shipped as silence. The loop is self-sustaining.

This was not the FW-64 callback overwrite, the unsafe AV/C background restart, or RX/TX SYT phase — those remain separate documented defects and were not required to produce this incident.

Changes

IsochTxDmaRing — true-cycle completion accounting (TX-LAPLOSS-001)

ComputeLostLapPackets() recovers true progress from the OUTPUT_LAST completion timestamps (the context transmits exactly one packet per cycle), detects whole lost laps — including the pointer-unmoved exactly-one-lap stall, which was previously invisible — and reconciles completionCursor, softwareFillAbsIdx_, and the completion stamps forward. The producer's committed lead (678 packets) already covers the reconciliation jump, so a stall degrades to a bounded ≤6 ms glitch instead of permanent time dilation. New lapLossEvents / lapLossPacketsTotal counters plus an IT LAP LOSS ring record make every event attributable.

IsochTransmitContext::Poll() — sustained IRQ-silence fault (bounds TX-IRQ-001)

The watchdog now logs its first engagement and, after 16 consecutive IRQ-silent kicks (~1 s), stops the context via the existing TX-fault path. Watchdog-carried streaming re-transmits stale ring laps between kicks and is never a valid steady state — a dead interrupt path now produces a clean, attributable fault within a second instead of a half-hour zombie.

PrepareTransmitSlots() — failure-class replay handling

Replay read failures are no longer uniformly "reset + re-arm alignment":

• overwritten → re-anchor the reader via Begin() and retry in place ([TxReplay] reclamped). Skipped entries only shift NODATA placement, which IEC 61883-6 blocking permits; DBC continuity is packetizer-owned and the SYT offset drifts sub-tick across the skipped span. No alignment re-arm.
• ahead → hold the reader and ship one NODATA; the same cursor reads successfully once RX catches up.
• Only a genuine RX-domain transition (epoch change, establishment loss, seqlock miss) resets the reader and re-arms frame-cursor alignment.

This removes the realign churn that orphaned host frames.

FCPTransport — accept response delivered before local write completion (separate commit)

AR Request drains before AR Response, so a target's FCP response can arrive before our own command-write acknowledgement. Dropping it made the Duet 48 kHz prepublish time out and deferred the audio nub. The response itself proves delivery; it is now accepted for the active write attempt, and the later local write callback is a harmless stale event.

Validation

• Host tests: 1383/1383 pass, including new suites: IsochTxLapLossMath.* (timestamp math incl. 8-second wrap and jitter tolerance), IsochTxDmaRingTest.RefillReconcilesLostLapForward, IsochTxDmaRingTest.RefillDetectsWholeLapStallWithUnmovedPointer, RxDrivenTimingTests.ReaderReBeginReanchorsAfterHistoryOverwritten, and the updated FCPTransportTests.AcceptsResponseBeforeCommandWriteCompletion.
• ./build.sh --no-bump succeeds (the 6 analyzer warnings are pre-existing).
• Hardware-validated on a live Duet run (~15+ min, clean audio): exactly one [TxAlign] at stream start, zero [TxReplay] failures after startup, zero payload-writer losses — and one real ~8.5 ms refill stall at ~3 min was caught and reconciled (IT LAP LOSS: ring lapped 48 packets … delta=20, committed − completion = 678) with no audible consequence and no downstream thrash. On the old code that exact event class started the death spiral.

Still open (documented in AVC_RECOVERY_AND_SYNC_ALGO_AND_BUGS.md)

• TX-IRQ-001 — why the IT interrupt path died mid-session. The new watchdog-engagement log will timestamp the next occurrence; the 16-kick fatal bounds the damage either way.
• FW-64 items (AUDIO-RECOVERY-ROUTING-001 callback overwrite, AVC-RECOVERY-002 unsafe background restart) are unchanged by this PR.
• Telemetry tuning: a healthy run currently rate-limit-logs [TxPrepFrame] frame=1 continuously because the 400-cycle exposure target is structurally unreachable within the 678-packet preparation limit (actual exposure lead ~27–35 ms, ~3× the safety floor — no frames lost). The frameShort gate should move to the real safety floor.

🤖 Generated with Claude Code

[mrmidi]

Hardware validation status — 2026-07-19 Duet endurance run

Live run on the Apogee Duet with this branch installed: 31m50s of continuous duplex streaming, audibly clean end to end. Verified from three independent sources.

FireBug wire capture (!duetok!.txt)

• 15,282,995 packets per channel — exactly 8000/s — with 0 silent cycles, 0 short packets, 0 lost.
• Channel 0 (host→Duet) carries real PCM the whole way (AM824 label 0x40 with a live waveform), where yesterday's !duet-fubar.txt capture showed all-zero payloads in the same transport-healthy shape.
• DBC advances +8 per DATA packet with no seam; blocking cadence D,D,D,N intact both directions; TX SYT tracks RX SYT cycle-for-cycle to ±1 tick (expected: the TX SYT is a re-anchored replay of the device's own phase offset, and the Duet is cycle master, so there is no second clock to drift).

MCP driver ring (zero dropped records)

The fix absorbed four organically-occurring lap-loss events — refill stalls of 6.9–10.8 ms at ~3, ~26, ~28, and ~31 minutes, each a single ring lap:

IT LAP LOSS: ring lapped 48 packets between refills delta=20 completion=1511844  committed=1512522
IT LAP LOSS: ring lapped 48 packets between refills delta=38 completion=12644508 committed=12645186
IT LAP LOSS: ring lapped 48 packets between refills delta=9  completion=13610976 committed=13611654
IT LAP LOSS: ring lapped 48 packets between refills delta=7  completion=14906564 committed=14907242

Every event reconciled with the producer at its full 678-packet committed lead (committed − completion = 678), so the fill jump stayed inside the committed window: no fault, no time-dilation debt, no audible consequence. On the pre-fix code, each of these events deposited a permanent 6 ms completion debt — this exact event class is what compounded into yesterday's 35-minute all-zero zombie.

Audio-side counters over the full run:

Signal	Count
[TxAlign] realignments after stream start	0 (one continuous host-frame mapping)
[TxReplay] fail=overwritten / reclamped	0 (reader never left its window)
[TxReplay] fail=ahead	4 (transient, absorbed in place — no reset, no realign)
[PayloadWriter] frame losses	0
Watchdog engagements (TX-IRQ-001)	0 (interrupt path alive all session)

Notes

• IT completion-delta high-water reached 43/48 packets between lap-loss events — this machine's scheduling load rides close to the 6 ms ring depth, so the reconciliation path is load-bearing, not theoretical.
• TX-IRQ-001 (the interrupt-path death that enabled yesterday's zombie) did not recur this session; if it does, the watchdog-engagement log will timestamp it and the 16-kick fatal bounds it to ~1 s.
• Known cosmetic issue (follow-up, not this PR): a healthy run continuously rate-limit-logs [TxPrepFrame] frame=1 because the 400-cycle exposure target is structurally unreachable within the 678-packet preparation limit; actual exposure lead is ~27–35 ms, ~3× the safety floor, and no frames are lost.

🤖 Generated with Claude Code

[mrmidi]

Status update — direction change, do not merge as-is

Hardware testing on a Saffire Pro 24 DSP (DICE) reversed the conclusion of the TX lap-loss work. Summary before this branch gets rebuilt.

The lap-loss detector was the regression, not the fix

On the Saffire the detector phantom-killed a healthy stream. Every IT LAP LOSS event reported a mod-48 delta below 48 (0,3,5,8,9,11,18,19,23,26,27,31,40) — the hardware never advanced a full ring between refills, so mod-48 accounting was correct throughout. The timestamp method was double-counting a stale completion baseline under coalesced interrupts and inventing laps; the delta=0 → "2832 packets" fatal is that bug run away. FireBug confirms the stream was continuous — 2,241,634 packets with real PCM — until our own fatal fired.

Reinforcing data point: DICE was stable before any of this — merge 1bbb70e (2026-07-14) had no lap-loss logic at all (pure mod-48) and streamed cleanly. The zombie appeared only after the 2026-07-18 receive-seam / FW-64 refactors. So the detector treats a symptom of TX-IRQ-001 (the Duet's dead interrupt path, where 68 ms watchdog gaps really do exceed the 6 ms ring) and breaks the healthy-coalescing case — and it's a boundary violation besides (timing policy inside the payload-opaque transport).

Real root cause: cross-backend leakage (dice ≠ avc ≠ bebob)

references/linux-sound-firewire-stack proves the boundary ASFW breaks:

• Recovery is per-family. Each of dice/ bebob/ oxfw/ fireworks/ motu/ … owns its *_stream.c with its own *_stream_update_duplex(). No shared recover_streaming.
• The shared core is family-blind. amdtp-stream.c/cmp.c/fcp.c have no backend branching — only device-quirk flags.
• One content callback; errors flow up as state. Backends inject process_ctx_payloads; on trouble the engine sets its own state and the family polls amdtp_streaming_error(s). The mechanism never dispatches a backend's recovery.

The Focusrite DICE kext corroborates from the Apple side: DICE recovery is driven by the DICE notification register (NotificationWriteCallback → RequestStreamingRestart), and adjustOutputPhase is a bounded phase servo that tolerates a standing RX/TX SYT offset — so the "SYT out of sync" capture was healthy, not a fault.

ASFW's leaks: a single shared IsochService::timingLossCallback_ (AV/C overwrites DICE, last-writer-wins), cross-backend RecoverStreaming off an RX-replay-reset signal common to both, and the lap-loss detector.

Plan for this branch

1. Revert the lap-loss reconciliation (ComputeLostLapPackets + Refill reconcile/context-stalled). Keep the sound, backend-agnostic pieces: failure-class replay handling (hold on ahead, no realign) and the watchdog-silence fatal + ctrl/intEvent snapshot.
2. Sever the shared timingLossCallback_; make transport report-only state the backends poll.
3. Re-home DICE recovery onto DICE notifications; keep AV/C separate.

TX-IRQ-001 (why Duet interrupts die) remains an independent open root cause. Full write-up in AVC_RECOVERY_AND_SYNC_ALGO_AND_BUGS.md (§ "backend boundary leakage", pushed in 940f02d).

🤖 Generated with Claude Code