Never-dark AI video middleware.

---

braidpipe ingests a live video stream, hands raw frames to a Python process for AI/CV work over shared memory, re-encodes the result, and streams it out — and if that Python process crashes, stalls, or falls behind, the output stream keeps running on untouched frames instead of going dark.

The Rust daemon owns the media path. Python only ever sees pixels in a shared-memory slot and a small JSON message telling it which slot to look at. That separation is the whole point: an exception in a model's inference code must never be able to take the broadcast off air.

                     ┌─────────────────────────────────┐
   SRT / UDP / RTP   │            braidpipe            │
 NDI / camera / file │                                 │
        ──────────►  │  decode ──┬── queue ─────────┐  │
                     │           │                  │  │
                     │           │            input-selector ──► encode ──►  RTMP / SRT
                     │           │                  │  │                     UDP / display
                     │           └── appsink ──┐  appsrc
                     │                         │     ▲  │
                     └─────────────────────────┼─────┼──┘
                                    shared memory + Unix datagrams
                                               ▼     │
                                        ┌────────────┴───┐
                                        │  Python worker │
                                        │  (numpy/cv2/…) │
                                        └────────────────┘

The input-selector decides, frame by frame, whether the viewer sees the AI branch or the raw passthrough branch. A watchdog flips it based on whether Python is actually keeping up.

Table of contents

• Table of contents
• How the failover works
• Requirements
• Install
• Quick start
• Real-world pipelines
• Output presets
• Audio passthrough
• Command-line reference
• Writing a Python worker
• The IPC contract
• Testing failover
• Measuring latency
• Monitoring
• Troubleshooting
• Project layout
• Development
• Known limitations
• Contributing
• License

How the failover works

Availability is enforced at two independent levels, so a single missed frame is handled differently from a dead worker.

Per frame (the relay). For every frame tapped off the pipeline, the relay writes it into a shared-memory slot, signals Python, and waits for an acknowledgement. The budget is 1.5 frame periods — 50 ms at 30 fps. If the ack doesn't arrive in time, or shared memory is full, or Python reports a failure, the relay pushes the original, unmodified frame downstream and reclaims the slot. Timing jitter therefore costs you an un-overlaid frame, not a gap in the stream.

Over time (the watchdog). The relay reports each success and failure to a health counter. Thirty consecutive failures — about one second at 30 fps — mark the worker unhealthy, and the watchdog switches the input-selector to the passthrough branch. One successful roundtrip resets the counter and the AI branch is selected again. The AI branch starts out unselected and has to earn its place with a first successful frame, so a worker that never starts correctly can't take frames with it.

Both branch queues are leaky=downstream, which matters more than it looks: without leaky queues, buffers piling up on the inactive selector pad eventually block the tee and stall the entire pipeline, including the branch that was working fine.

Requirements

Component	Version	Notes
Rust	1.85+	Edition 2024
Python	3.10+	3.13+ recommended, see note below
GStreamer	1.20+	Needed for appsrc leaky-type; developed against 1.28
OS	Linux or macOS	POSIX shared memory + Unix datagram sockets

GStreamer plugins depend on what you actually stream: srt for SRT, x264/libav for H.264, rtmp for RTMP output, avfvideosrc (macOS, in plugins-bad) or v4l2src (Linux, in plugins-good) for cameras, and a third-party plugin for NDI.

On Python 3.13+, SharedMemory(track=False) keeps Python's resource tracker from unlinking the Rust-owned segment when the worker exits. On older versions the bundled worker falls back automatically, but you may see a resource-tracker warning at shutdown and should restart the daemon rather than reusing the segment.

Install

Debian / Ubuntu:

sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \
                 gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
                 gstreamer1.0-plugins-ugly gstreamer1.0-libav

macOS (Homebrew):

brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav

Then build the workspace and set up the Python side:

git clone <your-fork-url> braidpipe
cd braidpipe
cargo build --release

python3 -m venv .venv
.venv/bin/pip install numpy opencv-python

The daemon prefers .venv/bin/python3 when that path exists and otherwise falls back to python3 on PATH, so a virtualenv at the repository root needs no extra configuration.

Quick start

Run from the repository root so the default worker path resolves:

cargo run -p braidpipe --release

That builds a 1280×720 test pattern, runs it through the bundled worker (which draws a frame counter onto each frame), and displays the result with autovideosink. You should see the overlay text updating on a moving ball.

To confirm the media path alone, with no Python involved:

cargo run -p braidpipe --release -- --passthrough-only

Real-world pipelines

SRT in, RTMP out — the common broadcast shape:

cargo run -p braidpipe --release -- \
  --uri 'srt://0.0.0.0:9000?mode=listener' \
  --sink 'videoconvert ! video/x-raw,format=I420 \
          ! x264enc tune=zerolatency bitrate=4000 speed-preset=veryfast key-int-max=60 \
          ! h264parse config-interval=-1 ! flvmux streamable=true \
          ! rtmp2sink sync=false location=rtmp://localhost/live/stream'

sync=false is not incidental — it is worth about 48 ms, for the reasons in Measuring latency.

NDI in, with a plugin that registers the ndi:// scheme:

cargo run -p braidpipe --release -- --uri 'ndi://Studio%20Camera' --sink 'videoconvert ! autovideosink'

Raw RTP in, which needs explicit caps and a depayloader — that's what --source is for:

cargo run -p braidpipe --release -- \
  --source 'udpsrc port=5000 caps="application/x-rtp,media=video,encoding-name=H264,payload=96" \
            ! rtph264depay ! h264parse ! decodebin3 ! videoconvert ! videoscale' \
  --sink 'videoconvert ! autovideosink'

A webcam, on macOS (avfvideosrc) or Linux (v4l2src device=/dev/video0):

cargo run -p braidpipe --release -- \
  --source 'avfvideosrc device-index=0 ! videoconvert ! videoscale ! video/x-raw,width=1280,height=720,framerate=30/1 ! videoconvert' \
  --sink 'videoconvert ! autovideosink'

The scaling matters: --width/--height fix the shared-memory slot geometry, and a camera negotiates whatever resolution it likes unless you tell it otherwise. List your devices with gst-device-monitor-1.0 Video/Source — the index is not always the one you expect, and on macOS index 0 is often an iPhone offering itself as a Continuity Camera. A camera that is present but not actually streaming sits in PLAYING and delivers nothing, which looks exactly like a braidpipe stall; check it in isolation first:

gst-launch-1.0 avfvideosrc device-index=0 num-buffers=10 ! fakesink

That should reach end-of-stream in a couple of seconds. If it hangs, the problem is the camera, not this project.

1080p60:

cargo run -p braidpipe --release -- --width 1920 --height 1080 --fps 60 \
  --source 'videotestsrc is-live=true pattern=ball ! video/x-raw,width=1920,height=1080,framerate=60/1 ! videoconvert' \
  --sink 'videoconvert ! autovideosink'

--uri and --source are mutually exclusive. Use --uri when GStreamer can figure out the source on its own (it picks srtsrc ! decodebin3 for srt:// and uridecodebin3 for everything else); use --source when you need to spell out elements yourself.

Output presets

Writing the sink by hand, as above, gives full control — but most deployments want one of a few well-understood points on the latency/bandwidth curve. --output plus --preset builds the whole encoder + mux + sink chain for you, the way ffmpeg's -preset expands into a bag of x264 options:

cargo run -p braidpipe --release -- \
  --uri 'srt://0.0.0.0:9000?mode=listener' \
  --preset lowlatency --output rtmp://localhost/live/stream

--output understands rtmp://, srt:// and udp://host:port, and picks the right mux for each (FLV for RTMP, MPEG-TS for SRT/UDP). The daemon logs the sink it built at startup, so you can copy it out and use it as a --sink starting point.

Preset	Encoder settings	GOP	VBV	Sink sync	SRT latency	Intent
zerolatency	ultrafast + zerolatency, 6000 kbps	1 s	100 ms	false	50 ms	Every latency lever pulled; bandwidth pays for it
lowlatency (default)	veryfast + zerolatency, 4500 kbps	2 s	200 ms	false	125 ms	The measured sweet spot — same ~40 ms p50 as the tuned harness sink
balanced	medium + zerolatency, 3000 kbps	2 s	500 ms	false	250 ms	Better compression, still no B-frame delay
bandwidth	slow, B-frames + lookahead, 1800 kbps	4 s	1000 ms	true	500 ms	Minimum bits for the quality; adds several frames of encoder delay by design

The VBV column is what makes the bitrate column mean something on the wire: it bounds how far above the target the encoder may burst, and how much encoded data a receiver has to be ready to buffer — so it is simultaneously a bandwidth cap and hidden latency. x264's own default (600 ms) would allow bursts more than half a second long.

Bitrates assume 720p30 — scale them for other formats. A preset only decides defaults; every parameter yields to an environment variable, so you can start from a profile and turn one knob:

Variable	Overrides	Values
BRAIDPIPE_ENCODER	encoder	x264 (default) or vtenc (macOS VideoToolbox)
BRAIDPIPE_BITRATE_KBPS	target bitrate	kbps
BRAIDPIPE_SPEED_PRESET	x264 speed preset	ultrafast … placebo
BRAIDPIPE_ZEROLATENCY	zero-latency tuning	1/0 — x264 tune=zerolatency, vtenc realtime
BRAIDPIPE_GOP_SECONDS	keyframe interval	seconds
BRAIDPIPE_VBV_BUF_MS	x264 VBV buffer (burst bound)	milliseconds
BRAIDPIPE_SINK_SYNC	sink clock sync	1/0 — see Measuring latency for why 0 is worth ~48 ms
BRAIDPIPE_SRT_LATENCY_MS	srtsink latency budget	milliseconds

# lowlatency profile, but cap the bandwidth
BRAIDPIPE_BITRATE_KBPS=2500 cargo run -p braidpipe --release -- \
  --preset lowlatency --output srt://127.0.0.1:8888

Verified end-to-end with the latency harness: --preset lowlatency --output rtmp://… measured 39.8 ms p50 / 45.6 ms p99 worker→receiver at 720p30, identical to the hand-tuned sink.

Measured bandwidth

Bitrate targets are promises until you look at the wire, so scripts/preset-bandwidth.sh runs each preset through the full daemon + worker path, captures the RTMP output, and reports what actually left the encoder. Content is a moving scene blended with 30% white noise — a stand-in for camera footage, hard enough to push rate control against its cap. 20-second runs at 720p30, startup excluded:

Preset	Target	Mean on wire	Worst 1 s	Worst 250 ms burst	Output fps
zerolatency	6000 kbps	5123	5332	6481	30.0
lowlatency	4500 kbps	4501	4525	4941	30.0
balanced	3000 kbps	3000	3033	3291	30.0
bandwidth	1800 kbps	64	142	439	30.0

Three things worth reading out of that table:

• lowlatency and balanced hold their targets to within 1%, and no preset's worst 250 ms burst exceeds its target by more than 10% — that is the VBV bound doing its job. Provision the link for the target plus ~10% and it will not be surprised.
• zerolatency runs ~15% under target on hard content. The 100 ms VBV is tight enough to constrain ABR itself, trading a little quality for the strictest burst bound. That is the correct trade for its use case.
• bandwidth spends almost nothing on this content, and that is by design, not a bug. Without tune=zerolatency, x264's mbtree lookahead rates every block by how much future frames can predict from it — and noise predicts nothing, so mbtree declines to encode it. The synthetic content is 30% noise; real footage has structure everywhere and will sit far closer to target. Either way the target is a hard ceiling, never a floor: this preset buys quality-per-bit, not constant bandwidth. (On pure noise — BRAIDPIPE_BW_PATTERN=snow — the effect is even starker, while the three zerolatency presets still hold their caps, since zerolatency disables mbtree.)

Audio passthrough

Real sources carry audio, and the output should too. --audio routes the source's audio around the AI branch — decoded, re-encoded to AAC, and joined back at the output muxer:

cargo run -p braidpipe --release -- \
  --uri 'srt://0.0.0.0:9000?mode=listener' \
  --audio --preset lowlatency --output rtmp://localhost/live/stream

How sync works. There is no dedicated sync machinery, because none is needed: the relay pushes every video frame back into the pipeline with its original PTS — whether the worker processed it or the deadline passed and it went through unchanged — and audio keeps the PTS the source gave it. The muxer pairs the two streams by timestamp, exactly as it would in a plain GStreamer pipeline. That also means failover cannot desynchronize anything: the input-selector switches video branches while audio never stops flowing.

Measured on a live SRT source (video + audio) relayed to RTMP at 720p30: steady-state packet cadence was exactly 33.33 ms for video and 21.33 ms for audio (1024 samples at 48 kHz), with under one video frame of relative drift over a 19 s run — both with the worker healthy and with a worker that missed every deadline.

--audio needs to know where audio comes from and where it goes:

• Source — with --uri this is automatic. With a custom --source, name your demuxer or decodebin decoder so the audio branch can tap it.
• Sink — with --output this is automatic (preset muxers are named). With a custom --sink, name your muxer mux.

Variable	Overrides	Default
BRAIDPIPE_AUDIO_ENCODER	AAC encoder element	avenc_aac (in gst-libav; fdkaacenc, faac also work)
BRAIDPIPE_AUDIO_BITRATE_KBPS	audio bitrate	128
BRAIDPIPE_AUDIO_BRANCH	the entire generated branch, verbatim	decoder. ! queue ! audio/x-raw ! audioconvert ! audioresample ! avenc_aac bitrate=128000 ! aacparse ! queue ! mux.

If the source has no audio stream, don't pass --audio — the audio branch would wait forever for a pad that never appears and GStreamer fails the pipeline with a delayed-linking error.

Command-line reference

Flag	Default	Purpose
-i, --source <PIPELINE>	test pattern	Explicit GStreamer source fragment
--uri <URI>	—	Input URI decoded by GStreamer (srt://, udp://, rtp://, ndi://, file://)
-o, --sink <PIPELINE>	videoconvert ! autovideosink	Output fragment appended after the selector
--output <URL>	—	Publish target (rtmp://, srt://, udp://host:port); builds the sink from --preset
--preset <NAME>	lowlatency	Latency/bandwidth profile for --output, see Output presets
--audio	off	Carry source audio to the output muxer, see Audio passthrough
-p, --python-script <PATH>	python/braidpipe/worker.py	Worker to launch
-f, --fps <N>	30	Frame rate; sets the relay deadline and watchdog tick
--width <N> / --height <N>	1280 / 720	Shared-memory slot geometry
--shm-name <NAME>	/braidpipe_buffer	POSIX shared memory object name
--rust-sock <PATH>	/tmp/braidpipe_rust.sock	Where the daemon listens for acks
--python-sock <PATH>	/tmp/braidpipe_python.sock	Where the worker listens for notifications
--passthrough-only	off	Media path only; no worker, no shared memory
--metrics-port <N>	9184	Prometheus endpoint on 127.0.0.1, see Monitoring; 0 disables
--metrics-drain-ms <N>	2000	How long to keep serving metrics after a shutdown signal, so the down state gets scraped

--width/--height must match the frames your source actually produces after videoscale, because they define the slot size that both sides index into.

Set RUST_LOG=debug to see per-frame relay activity, including dropped and stale acks.

Writing a Python worker

A worker is a loop over one Unix datagram socket. SharedMemoryManager gives you a zero-copy NumPy view of the slot, so mutating the array in place is writing to the output frame — there's no separate send step for pixels.

import json, os, socket
from shm import SharedMemoryManager

sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
if os.path.exists("/tmp/braidpipe_python.sock"):
    os.remove("/tmp/braidpipe_python.sock")
sock.bind("/tmp/braidpipe_python.sock")

shm = SharedMemoryManager()   # reads geometry from the header Rust wrote

while True:
    packet = json.loads(sock.recvfrom(512)[0])
    frame = shm.get_slot_numpy_array(packet["slot_index"])   # (H, W, 3) uint8, RGB

    # ... your inference here; mutate `frame` in place ...

    shm.mark_slot_free(packet["slot_index"])
    try:
        sock.sendto(json.dumps({
            "frame_id": packet["frame_id"],
            "slot_index": packet["slot_index"],
            "processing_time_us": 0,
            "success": True,
        }).encode(), "/tmp/braidpipe_rust.sock")
    except OSError:
        pass   # a full socket buffer is not worth dying over

Four rules keep the stream healthy:

1. Finish inside the budget. You have 1.5 frame periods. Slower than that and your output is simply not used for that frame — correctness is preserved, but the overlay flickers. For heavy models, drop the input --fps, or run inference on every Nth frame and cache the result.
2. Always free the slot. The ring has four slots; leaking them starves the relay. Free the slot even on your own error paths.
3. Always send an ack, and never die sending it. Report "success": false for a failed frame — the relay treats that as a failure and passes the original through, which is exactly right. Wrap the send in try/except OSError; a full datagram buffer (ENOBUFS) is normal backpressure, not a fatal condition.
4. Frames are RGB, not BGR. OpenCV's conventions assume BGR, so the familiar (0, 0, 255) "red" renders as blue here. Use (255, 0, 0) for red, or convert with cv2.cvtColor if you're feeding a model trained on BGR.

Point --python-script at your own file. Because Python puts the script's own directory on sys.path, a worker living beside shm.py can from shm import SharedMemoryManager directly; from elsewhere, add python/braidpipe to sys.path or install it as a package.

Bundled examples

Four workers ship with the project, each self-contained and runnable as-is:

Worker	Needs	Shows
worker.py	opencv	The minimum contract: a text overlay and a frame counter
worker_edges.py	opencv	A whole-frame pixel transform, and reporting success: false instead of dying
worker_detect.py	ultralytics, torch	A model too slow to run inline, moved to a thread with cached results
worker_stamp.py	numpy	Instrumentation rather than transform — see Measuring latency

cargo run -p braidpipe --release -- --python-script python/braidpipe/worker_edges.py
cargo run -p braidpipe --release -- --python-script python/braidpipe/worker_detect.py

worker_edges.py is the one to reach for when testing the plumbing: no model, no network, and it rewrites only the left half of the frame so the boundary between processed and untouched pixels is visible on screen.

worker_detect.py runs YOLO and is the more instructive one. A CPU inference pass does not fit in 50 ms, so the socket loop never waits for it — a detector thread takes a copy of every third frame, and every frame is annotated with the most recent boxes available. Boxes lag the picture slightly; no frame misses its deadline. It also caps torch.set_num_threads and the inference resolution, because torch left unbounded takes every core and starves the loop that only needs a millisecond of it. Without those caps the worker flaps between branches every few seconds; with them the AI branch stays selected. The first run downloads weights (~6 MB) into the working directory.

Writing a worker in another language

Nothing in the contract is Python-specific. crates/braidpipe-ipc/examples/worker.rs is a complete worker in Rust — it attaches to the segment, reuses the daemon's own ShmHeader/SlotHeader/packet types so it cannot drift out of sync with them, transforms pixels, frees the slot, and acks.

A worker only ever attaches to shared memory. Never call ShmRingBuffer::create from one: it unlinks and reinitialises the segment out from under the running daemon.

The daemon always spawns its --python-script, so point that at a no-op and run your own worker alongside it:

# terminal 1 — creates the shared memory segment and streams
cargo run -p braidpipe --release -- --python-script /dev/null

# terminal 2 — the AI branch is selected on this worker's first good frame
cargo run -p braidpipe-ipc --release --example worker

Any language with POSIX shared memory, Unix datagram sockets, and a JSON parser can do the same. What it must implement is the IPC contract below, in full — the four rules above apply regardless of language.

The IPC contract

Shared memory — one POSIX object holding a 32-byte header followed by slot_count slots. Each slot is a 24-byte header plus width × height × channels bytes of pixels. All layouts are explicitly padded on the Rust side and mirrored by struct format strings in python/braidpipe/shm.py, which assert their own sizes at import — a mismatch fails loudly instead of silently reading garbage.

Structure	Rust	Python format	Size
Segment header	ShmHeader	<IIBB2xI16s	32 B
Slot header	SlotHeader	<B7xQQ	24 B

Slot ownership is a single atomic state byte: FREE (0) → PROCESSING (2) claimed by Rust with a compare-and-swap, → READY_FOR_AI (1) once the pixels are written, and back to FREE by whichever side finishes — Python after processing, or Rust reclaiming a slot whose roundtrip failed. No locks, no ordering assumptions between processes beyond that byte.

Control channel — two Unix datagram sockets carrying one JSON object per message. Rust → Python announces a frame:

{"frame_id": 1081, "slot_index": 2, "timestamp_us": 1769506390123456}

Python → Rust acknowledges it:

{"frame_id": 1081, "slot_index": 2, "processing_time_us": 4210, "success": true}

Datagrams are used deliberately: they're unordered and droppable, which matches a real-time pipeline where a late frame has no value. The relay discards acks whose frame_id doesn't match the frame it's currently waiting on.

Testing failover

The interesting property is what happens when Python dies mid-stream. Start the daemon, wait for branch=AiProcess, then kill the worker using the PID from the log line Python worker active pid=…:

kill -9 <worker-pid>

Within about a second you should see Successfully switched video stream branch branch=Passthrough, the overlay disappear, and the stream continue without a stall, a black frame, or a dropped publisher connection. Measured on a 30 fps test pattern: the switch lands 1.0 s after the kill, and output frames keep arriving at exactly 30 fps across it.

Killing the worker by pattern is harder than it looks, and most obvious attempts are wrong.

pkill -f 'worker.py' matches the whole command line, and the daemon's own command line contains --python-script python/braidpipe/worker.py — so it kills the daemon too and proves nothing. Anchoring to the interpreter does not help either: .venv/bin/python3 is a symlink chain, so the running process reports its resolved interpreter path (…/Python.framework/Versions/3.14/Resources/Python.app/Contents/MacOS/Python on Homebrew macOS), and a pattern containing python3 never matches at all. It fails silently, which is worse than failing loudly.

Target the worker by parentage instead — it is the daemon's only child, on any platform:

pgrep -P "$(pgrep -x braidpipe)" -l     # check first: one PID, the worker
pkill -9 -P "$(pgrep -x braidpipe)"

Or just use the PID the daemon already printed, which is the same number.

Nothing respawns the worker (see Known limitations), but you can start one by hand and the daemon picks it up on its next successful frame — verified: branch=AiProcess returns within a couple of seconds:

.venv/bin/python3 python/braidpipe/worker_edges.py

Run it from the repository root, and note that a hand-started worker is no longer a child of the daemon, so the pgrep -P trick above will not find it a second time.

There's also a manual SRT check that exercises URI ingestion end to end with a graphical sink:

bash scripts/e2e-srt-autovideosink.sh

Measuring latency

bash scripts/rtmp-latency.sh

This publishes a test pattern over RTMP and reports how late every frame was when a receiver got it. It needs nothing installed beyond ffmpeg: -listen 1 makes ffmpeg the RTMP server braidpipe publishes to and the decoder, so no media server sits in the middle inflating the number.

There is no OCR and no guessing. worker_stamp.py writes the wall clock into every frame as a row of large black-and-white cells, and rtmp_latency_probe.py reads it back out of the decoded pixels and subtracts. Both processes are on one machine reading one clock, so this is a true one-way measurement rather than a halved round trip. Big cells are the point: H.264 will smear a thin line, but block-coded black and white survive any bitrate worth streaming — the runs below decoded 100% of frames.

Measured on an M-series Mac, 1280x720 @ 30 fps, x264enc tune=zerolatency speed-preset=ultrafast, medians over ~700 frames:

Leg	p50	What it covers
daemon → worker	0.17 ms	SHM write, UDS datagram, worker wake-up
worker → pipeline output	6.1 ms	SHM read-back, appsrc, videoconvert, sink
worker → RTMP receiver	39.5 ms	the above plus encode, flvmux, RTMP, demux, decode

So braidpipe's own contribution is about 6 ms, and everything else is the encoder and the transport. The IPC is not what costs you.

The single biggest knob is sync=false on the sink, worth 48 ms on its own:

Sink	p50
rtmp2sink (defaults)	89.0 ms
rtmp2sink sync=false	41.1 ms

A syncing sink holds each buffer until its running time plus the pipeline's configured latency, and processing-deadline alone contributes 20 ms of that by default. A live source already paces the pipeline, so the clock has nothing left to contribute — it only delays. Pinning video/x-raw,format=I420 before the encoder is worth another millisecond or so at the median, and stops the encoder inheriting 4:4:4 from the RGB the AI branch works in.

Hardware encoding is not automatically the low-latency choice here — VideoToolbox measured slower than x264 at ultrafast:

Encoder	p50
x264enc tune=zerolatency speed-preset=ultrafast	40.5 ms
vtenc_h264 realtime=true	46.5 ms

Useful knobs, all environment variables:

Variable	Default
BRAIDPIPE_RTMP_DURATION	30	seconds to measure
BRAIDPIPE_RTMP_ENCODER	x264	or vtenc
BRAIDPIPE_RTMP_TUNED	1	set 0 to reproduce the defaults row above
BRAIDPIPE_RTMP_SINK		replace the sink string outright
BRAIDPIPE_STAMP_BUSY_MS	0	give the worker a fake per-frame cost

That last one doubles as a failover test with numbers attached. A worker held 60 ms per frame is well past the 50 ms budget, and the run shows exactly what the design promises:

frames received : 402
  with barcode  : 0
  unreadable    : 402  (pre-keyframe, or passthrough frames)

worker->received: no samples
arrival interval  min= 16.30  p50= 33.33  p90= 35.49  p99= 37.87  max= 38.18  (ms, n=401)

Every stamped frame is gone — the AI branch never made its deadline once — and the output still arrives at a 33.33 ms median, which is 30 fps exactly. Nothing downstream could tell the worker had failed.

Monitoring

The daemon serves Prometheus metrics on http://127.0.0.1:9184/metrics (change with --metrics-port, 0 disables). Most of the numbers were already being measured for the failover logic — the endpoint makes them visible: every ack carries the worker's processing time, the relay times every roundtrip, the bridge counts the failure streak. The instrumentation adds nothing to the frame path beyond lock-free counter increments.

What you get, by the question it answers:

Question	Metrics
Is the AI output live right now?	braidpipe_last_ai_frame_timestamp_seconds, braidpipe_active_branch, braidpipe_worker_up
What's our availability?	braidpipe_branch_seconds_total{branch}, braidpipe_branch_switches_total{direction}
Is trouble coming?	braidpipe_queue_depth, braidpipe_shm_slots_occupied, braidpipe_failure_streak, braidpipe_stale_acks_total, roundtrip p99 vs braidpipe_relay_deadline_seconds
How fast is the worker?	braidpipe_roundtrip_seconds and braidpipe_worker_processing_seconds histograms
Is the stream healthy?	braidpipe_input_fps, braidpipe_pts_discontinuities_total, braidpipe_av_skew_seconds, braidpipe_keyframes_total, braidpipe_bus_messages_total
What's on the wire?	braidpipe_sink_bytes_total, braidpipe_output_frames_total, and full braidpipe_srt_* transport stats (RTT, loss, retransmits) when the pipeline has an SRT element
Are the processes healthy?	process_* for the daemon, braidpipe_worker_cpu_seconds_total / braidpipe_worker_resident_memory_bytes for the worker, braidpipe_worker_exits_total

The A/V skew gauge is the audio-sync claim from Audio passthrough, continuously verified in production: both streams' running time at the muxer, subtracted.

A ready-made Grafana stack lives in monitoring/:

cd monitoring && docker compose up -d
# Grafana: http://localhost:3000  (provisioned dashboard, no login)
# Prometheus: http://localhost:9090

It scrapes once a second and ships a provisioned dashboard (bandwidth, frame rates, latency percentiles against the deadline, branch state timeline, drops, backpressure, A/V skew, process usage, SRT transport) plus alert rules for the conditions worth paging on: daemon unreachable, worker down, stuck in passthrough, output dark, >5% deadline misses, stale AI frames, A/V skew over 100 ms.

Shutdown and stale panels

A metric that stops being scraped keeps its last value on screen, so a daemon that dies looks identical to one that is healthy and idle. Three things prevent that misreading:

• braidpipe_up goes to 0 the moment a shutdown signal arrives, and the endpoint stays open for --metrics-drain-ms (default 2000, ≥ one scrape interval) so the down state is actually recorded before the process exits.
• The dashboard gates on the scrape, not on the daemon's own metrics. The Daemon panel reads up{job="braidpipe"} — Prometheus writes that even when the target is gone — and the availability, worker, and fps panels are conditioned on it, so they blank or read DOWN rather than freezing on the last healthy sample. BraidpipeDown alerts on the same series, and is the only rule that can fire when the daemon no longer exists to report anything.
• Shutdown cannot hang. Ctrl+C reaches the worker only when both share a terminal, so the daemon SIGTERMs it explicitly (SIGKILL after 2 s). Pipeline teardown is bounded by a guard that force-exits --metrics-drain-ms + 4 s after the signal: on macOS, set_state(NULL) on the GL video sink deadlocks against its own GL thread, and a daemon wedged there is the worst case of all — still alive, still serving, still reporting the last healthy sample.

Troubleshooting

No output at all, or garbled/duplicated frames. Look for leftover daemons first — this is by far the most common cause. Old instances share the same socket paths, shared-memory name, and output URL, and they will happily fight over all three:

pgrep -fl braidpipe
pkill -9 -f 'target/release/braidpipe'

Symptoms of cross-talk include Discarded stale Python ack with wildly out-of-range frame IDs, and RTMP sinks failing to connect because another publisher holds the URL.

The pipeline reaches PLAYING and then nothing happens — no frames, no branch switch, near-zero CPU, no error on the bus. With a live source this is almost always a latency negotiation failure inside the pipeline. The appsrc declares max-latency=-1 to prevent it: an appsrc otherwise reports zero maximum latency while a live source reports a minimum of one frame period, the selector aggregates min > max, and the pipeline stalls silently. The two AI-side links (tee → appsink, and appsrc → selector) also carry their own videoconvert so the appsink's RGB requirement is never forced back through the source. If you see it again, the signature is in the GStreamer logs:

input-selector <sel:src>: minimum latency bigger than maximum latency

Raising GST_DEBUG to find it is reasonable, but redirect to a file you are willing to lose — that one error repeats per latency query and can produce gigabytes per minute.

Output goes dark when the AI branch is selected. Check the log for pipeline errors — the bus watcher surfaces asynchronous failures that would otherwise be silent. Then confirm your sink can accept the AI branch's caps, which are video/x-raw,format=RGB at the configured resolution.

Failed to set SHM size with ftruncate. A stale segment is still mapped by another process. create() unlinks before opening, which handles the usual case; if it persists, kill every braidpipe and worker process and try again.

Worker exits with OSError: [Errno 55/105] No buffer space available. The ack socket buffer filled up. The bundled worker catches this; a custom worker must too.

Python failed to respond within target deadline repeating. Inference is slower than 1.5 frame periods. Lower --fps, shrink the resolution, or process every Nth frame.

Overlay colours look wrong. Frames are RGB; OpenCV colour tuples are BGR. Swap the outer channels.

Project layout

Ports and adapters, so the availability logic can be tested without GStreamer or Python in the loop:

Path	Contents
crates/braidpipe-core/	The watchdog FSM and the StreamController / ShmWriter / AiBridge port traits. No GStreamer, no sockets.
crates/braidpipe-engine/	GStreamer adapter: pipeline construction, branch switching, bus error reporting, and the macOS run-loop wrapper.
crates/braidpipe-ipc/	The shared-memory ring buffer and the Unix-datagram control bridge with its health tracking, plus examples/worker.rs — a worker written in Rust.
crates/braidpipe/	The daemon: CLI, wiring, worker supervision, preset.rs — the latency/bandwidth profiles — and relay.rs — the appsink → shm → Python → appsrc data path.
python/braidpipe/	shm.py (the Rust layout mirror), stamp.py (the latency barcode), and four example workers: text overlay, edge transform, YOLO detection, and clock stamping.
scripts/	Manual end-to-end checks, the latency harness, and the per-preset bandwidth measurement.
monitoring/	Prometheus + Grafana compose stack: scrape config, alert rules, provisioned dashboard.
assets/	Logo files: transparent wordmark and icon PNGs, plus a multi-size .ico.

Development

cargo test --workspace
cargo clippy --workspace --all-targets
cargo fmt --all

relay.rs is the place to start reading if you want to understand or change frame handling: it's short, and every failure path in it exists to protect the never-dark guarantee.

Known limitations

• No worker respawn. If the Python process dies, the daemon logs it and stays in passthrough for the rest of the run. Restart the worker manually or supervise it externally.
• Single video stream. One source, one sink, one worker per daemon. Run multiple daemons with distinct --shm-name and socket paths for multiple streams.
• Full-frame RGB only. The alpha-overlay compositing path — where Python returns just a mask to be blended, instead of a whole frame — is not implemented yet.
• Frames are copied, not zero-copy, on the Rust side. Each frame is copied out of the GStreamer buffer into shared memory and back. Python's view is genuinely zero-copy; Rust's is not.
• No hardware-accelerated decode by default. decodebin3 picks whatever is available; wire an explicit hardware decoder through --source if you need one.
• Video only. Audio is not carried through the pipeline.
• No Python SDK. shm.py mirrors the shared-memory layout and nothing more: it is not packaged, not on PyPI, and has no importable name. Every worker re-implements the same socket loop, slot release, and ack handling by copying an example. A thin braidpipe package wrapping that loop would remove the copy-paste, and is the obvious next piece of work.

Contributing

Issues and pull requests are welcome. Please run cargo test --workspace, cargo clippy --workspace --all-targets, and cargo fmt --all before opening a PR, and describe what you tested — for media changes, say which source and sink you actually ran, since plugin availability varies a lot between machines.

Commit messages follow Conventional Commits with a single-line subject, for example fix(ipc): align shm layout with python.

License

Apache-2.0. See LICENSE.