YUV422 pipeline, 720p/1080p modes, tablet HID, USB virtual media + serial console, WiFi/WireGuard, new web UI - #1
Open
DatanoiseTV wants to merge 40 commits into
Conversation
Video: Kconfig-selectable capture pipeline. RGB888 (default, JPEG 4:2:0) or YUV422 via TC358743 UYVY output with a BitScrambler DMA loopback pass rotating each 32-bit word to the YVYU order the P4 JPEG encoder requires (the CSI-bridge color converter needs rev >= 3.0 silicon; this target is rev 1.x). All format-dependent constants derive from one descriptor block in capture_priv.h. The per-frame 6 MB cache msync in the encode loop is replaced by a one-time writeback+invalidate after allocation: the ring is DMA-only and the JPEG/BitScrambler drivers sync their buffers per run (invariant documented at the ctx definition). Test pattern (P4KVM_TEST_PATTERN): CPU-generated animated color bars in the native pixel format, published through the same ring/semaphore contract, so the whole downstream stack runs without the HDMI-CSI module. HID: add an absolute-pointer report (ID 3, 0..32767 axes) so tablet mode positions the host cursor 1:1 without drift; drop the abs-to-rel emulation. Relative mouse and boot keyboard unchanged. ATX: optional power/reset front-panel control via optocoupler GPIOs; 300 ms tap or 5 s force-off hold, per-pin busy guard, POST /atx. HTTP: TCP_NODELAY on stream and input sockets (multipart trailer chunks no longer wait on delayed ACKs), /stats JSON with per-stage pipeline timings, optional Basic auth on all routes - /ws gated via ws_pre_handshake_cb because esp_http_server completes the 101 upgrade before the URI handler runs. Recovery: skip HPD cycling while DDC5V is absent (slow 60 s retry in case the bit reads low transiently), escalate to a full TC358743 register re-init after three failed hotplug recoveries. All four configurations (default, yuv422, auth+atx+yuv422, testpat) build against ESP-IDF 6.0. Not yet validated on hardware; see README verification status.
The ESP32-P4 has no radio; boards like the P4-nano carry an ESP32-C6 over SDIO. esp_wifi_remote + esp_hosted proxy the standard esp_wifi API to it, so the STA code is plain esp_wifi with auto-reconnect. Modem power save is disabled: a KVM cares about input latency more than milliwatts. mDNS and netif/event-loop setup move out of ethernet.c into net_common.c so they run for either interface; Ethernet and WiFi can be up simultaneously. Off by default (P4KVM_WIFI_ENABLE); SSID/password in menuconfig, SDIO pinning in esp_hosted's own Kconfig section. Builds verified for the default (WiFi off) and sdkconfig.ci.wifi configurations.
Tablet (absolute pointer) becomes the default input mode: the host cursor tracks the browser pointer 1:1 with no pointer lock; clicking the video captures the keyboard (Esc is forwarded since there is no lock to escape). Relative pointer-lock mode stays for BIOS/UEFI and games that ignore tablet devices. The MJPEG client is rewritten around an incremental multipart parser that writes each JPEG byte exactly once into its final buffer (the old parser re-scanned and re-copied the whole accumulation buffer on every network chunk) and a decode loop that drops stale frames instead of queueing them. Parser covered by a host-side test (1-byte chunk splits, malformed-part resync, binary bodies containing CRLFCRLF). Adds: Ctrl+Alt+Del button, fullscreen, stats overlay (client draw fps + link Mbps + device /stats polled while visible), ATX power/reset/force- off buttons with confirmations (shown only when the firmware reports the GPIOs wired, probed with retries so a slow boot cannot hide them), staggered startup requests to avoid the lwIP connect burst, numpad and PrintScreen/ScrollLock/Pause key mapping.
README rewritten around the new feature set with explicit hardware constraints (rev 1.x silicon limits, 100 Mbit/s Ethernet ceiling, 2-lane MIPI budget, WiFi bitrate expectations) and an honest verification-status section: what is build- and host-test-verified versus what still needs on-hardware validation. CHANGELOG started at 0.2.0 (Keep-a-Changelog format). Ignore build_*/ variant directories.
IDF 6.0 defaults the P4 to rev >= 3.01, and rev <3 / >=3 images are mutually incompatible silicon targets. On the rev 1.3 board a rev-3-targeted image boot-loops with an immediate illegal-instruction panic before any bootloader output. The README already required selecting rev <3 in menuconfig; encode it in the defaults so a fresh idf.py set-target cannot silently produce a non-booting image.
New from-scratch shell: broadcast-monitor aesthetic (graphite, single amber signal accent, monospace telemetry, hairline panels), a proper no-signal state - animated static, decoded reason (DDC 5V absent / no TMDS / no sync / stream offline / device unreachable) and a fix hint - plus amber crop-mark brackets while the keyboard is captured. Right-hand drawer replaces the settings dropdown: keyboard helpers, host power, live JPEG-quality slider, pointer sensitivity, and a diagnostics table (stream/input/USB HID state, decoded HDMI source, pipeline, capture/encode fps and times, JPEG size, link rate, viewers, recoveries, encoder errors, heap/PSRAM, uptime, IP addresses, version). /stats gains usb_hid, ip_eth/ip_wifi, hostname and version fields for the panel. tc358743_sys_status now propagates I2C errors instead of returning garbage when the bridge is absent - the module-missing case now reads as 'device unreachable/no source' rather than random status bits. The proven engines (multipart parser, frame-dropping painter, HID queues, WS lifecycle, startup stagger) carry over unchanged; parser host tests still pass. Bundle: 32 KB single file.
First hardware run showed every reorder pass writing 4147196 of 4147200 bytes: with 64-bit prefetch and 32-bit consumption the downstream trailing counter starts before the final word drains, so trailing_bytes 8 left one 32-bit word in the pipe and the strict length check then discarded every frame (stream stayed black). Budget 16 trailing bytes - surplus zeros fall past the output DMA descriptor list and are clipped. Also throttle the failure log (it spammed the UART at frame rate) and count drops in enc_errors so /stats surfaces them.
Second hardware finding: with trailing_bytes 16 all 4147200 frame bytes arrive but the run returns ESP_ERR_TIMEOUT - the surplus trailing zeros have no output descriptor to land in, so the engine never raises EOF. The pipeline evidently holds 96 bits in flight (64-bit prefetch plus one 32-bit read stage), not 64. Pad the output buffer by 64 bytes and accept written >= frame_bytes; the encoder still reads exactly frame_bytes.
Hardware verification with the color-bar test pattern settled it: every bar decoded as if the encoder reads chroma-first, meaning IDF's JPEG_ENCODE_IN_FORMAT_YUV422 == FOURCC 'YVYU' names the little-endian 32-bit WORD value, not the byte sequence. Byte-wise the encoder wants U Y V Y - exactly the TC358743's native UYVY stream. The reorder pass was built on the opposite reading of jpeg_types.h and produced swapped chroma; it also measured ~28 MB/s (147 ms per 1080p frame, capping the pipeline at 5.6 fps), so it was doubly untenable. The YUV422 pipeline now feeds the framebuffer straight to the encoder (measured encode: 29 ms/frame at 1080p, 360 MHz). Kconfig symbol renamed P4KVM_PIPELINE_YUV422_BS -> P4KVM_PIPELINE_YUV422; the bsasm program stays in-tree as reference with the pipeline-depth findings documented (96 bits in flight, trailing-flush behavior).
WireGuard (P4KVM_WG_ENABLE): vendored trombik/esp_wireguard 0.9.0 into components/esp_wireguard with three IDF 6 patches - mbedtls 4 removed mbedtls/entropy.h, so the RNG bootstrap (CTR-DRBG layered on an entropy source that itself wrapped esp_fill_random) now draws from the hardware TRNG directly; two GCC 15 -Werror suppressions for a stringop-overread false positive in x25519 and the deliberately unterminated WireGuard protocol constants. The supervisor task waits for an IP, syncs NTP time (handshakes need wall-clock), connects, and re-handshakes after 30 s of stale peer. Keys/endpoint via menuconfig; tunnel state exposed as 'wg' in /stats and a WIREGUARD row in the diagnostics panel. WiFi reliability: reconnect moved off the event-loop task (the handler previously slept 1 s inside the loop) onto a one-shot esp_timer with exponential backoff 0.5-8 s, reset on every got-IP; all-channel scan selecting the strongest BSS (repeater/mesh setups); in-supplicant failure_retry_cnt=3; power save re-asserted WIFI_PS_NONE after every association since some coprocessor firmwares reset it on reconnect. Builds verified for the default (WG off) and sdkconfig.ci.wg configs.
Capture resolution becomes a boot-time mode persisted in NVS instead of a compile-time constant. Each mode carries its own EDID so the HDMI source actually outputs that timing: the 720p60 EDID is generated from the 1080p30 template (canonical CEA VIC-4 DTD, native SVD, checksums recomputed and parse-verified programmatically). Switching - drawer Video section or POST /video-mode - restarts the device, which a resolution change requires anyway (new EDID negotiation, re-sized DMA buffers). 720p60 is the new default: ~1/3 the MJPEG bitrate of 1080p at equal quality, matching what a 2.4 GHz WiFi link sustains (the C6 radio tops out around 20-40 Mb/s TCP; 1080p q70 at ~60-70 Mb/s is why fps sawtooths on WiFi while the encoder holds a measured steady 33 fps). HID absolute coordinates are now resolution-independent: the browser scales to the 0..32767 HID logical range itself and the firmware passes them through, so pointer mapping never depends on the active mode. The web canvas tracks the device-reported frame size from /stats.
Same approach as h2c-rpi: the TC358743 already emits the HDMI source's audio on its I2S pads (this firmware configures that path; the bridge is I2S master with clocks derived from the stream). Three jumper wires bring BCK/LRCK/DATA to P4 GPIOs set in menuconfig; the P4 receives as I2S slave (32-bit slots, top 16 bits kept) and streams raw PCM S16LE 48 kHz stereo over the /audio WebSocket in 10 ms frames (~1.5 Mbit/s), single listener, auth-gated like /ws. The UI gains a speaker toggle (shown only when the firmware reports audio wired) feeding an AudioWorklet ring buffer (~250 ms, silence on underflow), plus an AUDIO diagnostics row and an 'audio' field in /stats. Off by default; 48 kHz is assumed (the only rate HDMI mandates and the only one the EDID advertises). Compile-verified for enabled and disabled configs; hardware-unverified until the I2S pads are wired.
The stock config assumed 2 MB flash (bootloader warned every boot) and the single-app layout capped the factory partition at 1 MB - the app was at 95 % of it. Custom partition table with a 4 MB factory slot on the P4-nano's 16 MB flash, leaving room to grow and for a future OTA split. Also provide an explicit TinyUSB device descriptor (Espressif VID, generic TinyUSB PID, our existing strings) instead of the NULL that made the stack log fallback warnings at boot.
New runtime_cfg module: NVS-backed settings with Kconfig fallbacks, served by GET/POST /config (cJSON; moved to the registry component in IDF 6). Secrets - WiFi password, WireGuard private and preshared key - are write-only: reads return only configured/not-configured flags. Consumers (wifi_net, wireguard_net, atx_ctrl) read the store once at init, so the UI saves and restarts via the shared deferred-restart path. The drawer gains a Setup section (WiFi credentials, WireGuard endpoint, port, keys, tunnel IP, ATX GPIOs and polarity) with save-and-restart, and the Host Power section is now always visible - when the GPIOs are not configured it explains how to wire and set them instead of hiding. WiFi and WireGuard support are now compiled in by default (activation is a runtime decision: empty SSID or missing keys simply skip startup), so a stock build can be fully configured from the browser.
A stale sdkconfig kept CONFIG_P4KVM_WG_ENABLE unset from before the Kconfig default flipped to y, so the build silently dropped WireGuard and the tunnel state was always "off" regardless of pushed config. Pin WIFI/ETH/WG on here so a pre-existing sdkconfig can never build a core feature out again.
Toggleable HUD (default on) pinned to the stage corner, showing the live pointer report (absolute wire coords + pixel position, or relative deltas, plus button/wheel state) and keyboard activity (held keys with modifiers and a rolling list of recent keystrokes). Reads straight from the HID send path so it reflects what actually goes on the wire, and updates even when the input link is down. The no-signal overlay is now pointer-transparent so the HUD tracks the pointer on the test screen too. Lets you debug input without a host or serial console.
esp_wireguard registers a raw lwIP netif and stores its own device struct in netif->state. With LWIP_ESP_NETIF_DATA=0 (the default), esp_netif also uses netif->state for its back-pointer, so its global ext-callback (esp_netif_internal_dhcpc_cb) reinterpreted the WireGuard device as an esp_netif_t and null-dereferenced ip_info the instant the tunnel address was set -> Load access fault boot loop right after 'allowed_ip'. lwipopts.h only moves esp_netif's pointer into netif->client_data (freeing netif->state for third parties) when the bridge or PPP option is on; enable the bridge purely for that side effect. Verified on hardware (rev 1.3): boots clean past allowed_ip and reaches 'peer up'.
Present a removable USB mass-storage LUN to the target alongside the HID keyboard/mouse, backed by a PSRAM ramdisk. Boots as an empty formatted FAT12 floppy; an arbitrary .img/.iso up to 12 MiB can be streamed in from the browser and mounted, so the target can boot rescue/installer images without physical media. - main/usb_msc.c/.h: dynamic-capacity ramdisk with begin/load/commit image flow. A deferred UNIT ATTENTION (0x28/0x00) on remount forces the host to re-read the new capacity. Bounds-checked read10/write10, eject via START_STOP_UNIT, write-protect toggle. - main/usb_hid.c: composite descriptor now advertises HID (itf 0) + MSC (itf 1). Separate FS (bulk 64) and HS (bulk 512) config descriptors as the high-speed spec requires; usb_msc_init() runs before the driver install. - main/http_server.c: GET /media/status, POST /media/image (raw body, ?writable=), POST /media/eject. The image streams from the socket straight into PSRAM in 4 KiB chunks with no full-image bounce buffer. - WebUI: Virtual Media panel with file picker, writable toggle, XHR upload progress, mount/eject and live status. - components/espressif__esp_tinyusb: vendored from managed 2.1.1 with the six tud_msc_* callbacks weakened so the app's strong raw-block definitions win (esp_tinyusb only ships FAT-on-flash/SD backends). See PATCHES.md. Config: CONFIG_TINYUSB_MSC_ENABLED, BUFSIZE 8192, pinned in sdkconfig.defaults.
Expose a virtual serial port to the target as a third USB function (HID+MSC+CDC) and bridge it to a browser terminal, turning the KVM into a serial-over-LAN console for headless boxes, kernel panics, and BIOS/U-Boot serial menus. - main/usb_serial.c/.h: CDC-ACM port via esp_tinyusb's CDC API. Target->browser bytes stage through an 8 KiB stream buffer drained by a pump task that sends WS frames; browser keystrokes are written straight to the target. Single terminal client, newest wins, dropped from the httpd close hook. Non-blocking throughout so a stalled browser never back-pressures the target's console. - main/usb_hid.c: composite descriptor gains the CDC IAD+comm+data interfaces (itf 2/3, EP 0x83 notif, 0x04/0x84 bulk), guarded by CONFIG_TINYUSB_CDC_ENABLED so it cleanly falls back to HID+MSC when disabled. FS/HS share one CDC fragment macro differing only in bulk max-packet size. - main/http_server.c: /serial WebSocket (auth pre-handshake like /ws), serial state added to /stats. - WebUI: KVM/TERM view toggle in the top bar; TERM shows an xterm.js terminal wired to /serial, connecting only while the terminal is open. Terminal dot mirrors the target's DTR from /stats. - README: virtual-media and serial-console sections, including a Linux target guide (ModemManager udev ignore rule by VID/PID, serial-getty and console=ttyACM0 setup) so the console works without breaking things. Both features build and link but are hardware-unverified (port busy this session); flagged as such in the README.
max_uri_handlers was 12 but 15 handlers register once MSC + serial were
added, so httpd silently dropped /ws and /serial ("no slots left"). The
browser's WebSocket reconnect storm against the missing /ws then exhausted
the lwIP socket pool (accept -> ENFILE, error 23) - the long-hunted "HTTP
wedge", now confirmed from the boot log. Raise to 18 with headroom, and
document the count so future handlers bump it.
Also remove the temporary socket-diagnostic task now that the cause is found.
WebUI: virtual media moves out of the settings drawer into a top-bar disc
icon + popover, so the mounted image and mount/eject are one click away
without opening settings. The icon glows amber while an image is mounted;
the popover shows live status and upload progress.
Even with all URI handlers registered, the browser re-fetches /stream (and reconnects /ws, /serial) on any hiccup, and lwIP's default 60s MSL held each closed socket in TIME_WAIT for a full minute. A few reconnects per minute then filled the 24-socket pool and httpd accept() returned ENFILE (errno 23) - the same wedge symptom, now from socket lifetime rather than handler slots. It took ~8 min to accumulate instead of ~30s. Drop CONFIG_LWIP_TCP_MSL 60000 -> 5000 (ample on a LAN/tunnel) so churned sockets free almost immediately, and raise CONFIG_LWIP_MAX_ACTIVE_TCP 16 -> 24 to match the socket pool and give the httpd sessions headroom.
DatanoiseTV
changed the title
New "Max framerate" slider (1-60, default 30, persisted in NVS) caps the MJPEG send rate per viewer via GET /stream-fps?fps=N; reported as max_fps in /stats. The stream worker drops frames that arrive sooner than the target interval instead of sending them. On top of the cap, each worker adapts to its own link: it times every send and, when a send eats most of the frame budget (a back-pressured socket - slow client or a thin WireGuard tunnel), raises that viewer's interval toward a 1 fps floor; a fast send eases it back to the cap. This keeps a congested stream alive at a lower rate instead of blocking until send_wait_timeout and forcing the browser to re-fetch /stream - the reconnect churn that was pumping the socket pool.
HDMI is digital, so a static screen encodes to byte-identical JPEGs. After each encode, fingerprint the JPEG (length + CRC32) and skip publishing when it matches the last transmitted frame: an idle desktop now sends effectively nothing, while a changing region streams up to the per-viewer cap. front_idx stays on the last distinct frame, and a newly-connected viewer is sent the current frame immediately (worker seeds last_seq to seq-1) so a static screen is not blank on connect. Adds tx_fps to /stats (transmitted/changed rate, <= enc_fps when dedup drops static frames) and a STREAM RATE row in the diagnostics so the saving is visible. Note: the built-in test pattern animates every frame, so tx_fps tracks the capture rate there; the dedup only shows its effect against a real (static) HDMI source.
|
I did a quick test on your branch... It kinda works.. but I see 1 frame screen every 10 seconds for a very short period. I tried wifi and ethernet (direct p2p connection). |
Author
|
thanks for the report. i received my hdmi to csi adaptor so i will be debugging tomorrow |
|
Let me know if you want me to do more testing! Thanks! for your effort and patch, it looks very promising!!! (typing this 'via' the original firmware, that works well @20fps, forgot to mention that above. ) I also had to disable and remove the wireguard component as it failed to compile for me on esp-idf 5.5. (using Guition JC-ESP32P4-M3 DEV Board) |
I also tried the test pattern.. that worked fine. |
|
Ok, reading through your commits.. I found the issue.. I have very static screen.. it only updates every minute.. so either removing the 'changed' check, or just playing a video seems to fix it :-P |
|
The cacheline flush is not needed by the looks of it. |
This reverts commit 15d8385.
Author
|
Ha, you beat me to it, and you're right, that's not a bug at all. Content dedup only transmits a frame when the JPEG actually changes, so a desktop that repaints once a minute really does stream ~0 fps until something moves. Playing a video or wiggling the mouse proves the path is fine. Nothing to fix there. Given that, the cacheline invalidate I added was chasing a ghost, so I reverted it, you were spot on that it isn't needed. The original "invalidate once at alloc, never touch the ring from the CPU again" invariant holds on both our boards. Good to hear mouse/keyboard is solid over the OTG port. And thanks for the ESP-IDF 6 datapoint plus the ICG issue link, that sleep-clock ICG alloc failure looks like a general P4-on-6.0 thing rather than anything in this branch. Disabling ICG to get past it is a reasonable workaround for now. If you ever get a spare minute, the USB virtual media is still the one untested piece from your side: drop a small bootable .img/.iso via the disc icon in the top bar and see if the target enumerates it as a removable drive (native OTG port, same as the HID). But honestly you've already done more than enough here, really appreciate it. |
|
On mouse: In tablet mode, clicking does not work for me, it does in relative mode. It is kinda a bug though, I need to see the screen, even if does not changes. Showing it for ~1 second and then back to a static screen saying no frame received (dunno exact text) is not very usefull. |
|
I see this 98% of the time: 
|
The ESP32-P4 has a dedicated H.264 baseline encoder, separate silicon from the JPEG codec, so it can run alongside the MJPEG path on the same CSI frames. capture_h264 feeds the TC358743's native UYVY straight into the hardware encoder (ESP_H264_RAW_FMT_UYVY, no colour conversion) and hands Annex-B access units to a registered sink. Encoding is demand- gated: with no sink (no WebRTC viewer) the per-frame hook is a no-op and the encoder is torn down, so idle costs nothing and the MJPEG path is unaffected. Adds the espressif/esp_h264 and espressif/esp_peer managed components and the mbedtls DTLS-SRTP options esp_peer needs. Gated behind P4KVM_WEBRTC_ENABLE (depends on the YUV422 pipeline). esp_h264 1.3.6 calls esp_efuse_is_flash_encryption_enabled(), which IDF 6.0.0 does not ship; h264_efuse_compat.h is force-included into the component to forward it to esp_flash_encryption_enabled(). 720p is the clean target (16-aligned); 1080p (1080 % 16 != 0) is rejected until a padded capture is added. This commit builds the encoder only; the WebRTC session and signaling follow.
Author
|
Ok, I have an idea:
Will commit later so you can test. |
webrtc_kvm drives esp_peer as the answerer (controlled role): one viewer
at a time, send-only hardware-H.264 video (fed from capture_h264's sink)
and a reliable SCTP data channel carrying keyboard/mouse on the same peer
connection. Data-channel reports use the identical wire format as /ws, so
both paths funnel through the new usb_hid_dispatch_report(); the /ws
handler is refactored onto it.
Signaling is automatic over the device's own HTTP server: POST
/webrtc/offer takes the browser's offer SDP and returns
{"sdp":<answer>,"candidates":[...]} for setRemoteDescription +
addIceCandidate - one round trip, no external server. On the data channel
opening, the video sink is registered and an IDR is forced so a static
screen paints immediately instead of staying black.
All gated behind P4KVM_WEBRTC_ENABLE; stubs when off. Backend only - the
browser client follows.
|
For me having a constant stream is not a big issue, but I guess adaptive would be nicer. side note: I get around 9-10fps (17 with original firmware).. If I have some more time I will test a bit further. Thanks again for your patch/code.. This is now already a pretty functional KVM! Good use for me buying to many p4 boards :D. |
The web UI now upgrades to WebRTC after the MJPEG baseline is up: it POSTs an offer to /webrtc/offer, applies the answer + candidates, and on the H.264 track going live shows a <video> behind the (now transparent) canvas, which stays the input surface, and stops the MJPEG fetch. Keyboard/mouse route over the "hid" data channel via a hidSend() shim (data channel when open, else /ws), so all HID gates accept either transport. If the peer connection fails at any point, HID falls back to /ws and MJPEG resumes - nothing shows black and the KVM keeps working. MJPEG paints from boot so there is never a black screen while WebRTC negotiates; a TRANSPORT diagnostics row shows which path is live. Append ?nowebrtc=1 to force the MJPEG path.
Author
|
I think WebRTC will really add value and I was thinking if it would make sense to also add programmatic access - so a script can send keystrokes to a remote machine, reboot it and maybe some kind of fleet management. |
Author
|
Considering making a custom board with P4, ethernet, wifi and the HDMI-to-CSI2 bridge IC. |
Adds "webrtc" (off/idle/connecting/connected) and "h264_fps" to the /stats JSON, and surfaces the live H.264 frame rate in the TRANSPORT diagnostics row while a WebRTC viewer is connected.
|
I would def. be interested in a custom board. |
Author
|
I am getting now 33 FPS with the test pattern. |
First hardware run: signaling and SDP worked, but the ICE agent never
paired and the connection died after ~5 s, looping. Two causes from the
device log: esp_peer's agent was given no ICE server (cfg.server_lists
NULL), so it never properly gathered/bound a candidate; and the browser's
~12 candidates overflowed esp_peer's default cap of 10 ("Remote candidate
over limited 10").
Configure a public STUN server and raise max_candidates to 24 via
extra_cfg on both the device and the browser. On the same LAN the host
candidate still carries the connection - the ICE server is what makes the
agent gather it (and gives off-LAN/tunnel viewers a srflx candidate).
Widen the device candidate-settle window and the browser's connect
timeout to fit the STUN round-trip, and cap browser retries so a network
where WebRTC can't establish settles on MJPEG instead of renegotiating
forever.
|
on full hd, it sais '15.4fps' encode in side menu. Switching to 720p and low Q jpeg does not change the fps shown: |
The HID worker coalesces queued mouse messages, but merge_mouse_msgs also merged across button transitions (acc.buttons = add.buttons). A point-and-click sends press(btn=1) then release(btn=0) with no motion between; both land in the queue together and merge into a single report with buttons=0, so the press is erased and the host sees no click. Tablet mode hit this on every click because absolute positioning means you don't move the mouse to aim; relative mode masked it because aiming drags the pointer and spaces the reports apart. Flush the accumulator on any button change, exactly like the existing abs/rel mode-switch case, so every transition survives as its own report.
Content-dedup only transmits a frame when the JPEG changes, so a static desktop legitimately stops producing frames. The overlay treated "no frame for 2.5 s" as signal loss and covered a perfectly valid last frame after ~1 s, which the tester flagged as unusable. Only fall back to the NO SIGNAL overlay when the stale frame coincides with an actual fault: no frame ever received, stream offline, device unreachable, or the source lost lock (which /stats reflects within ~1 s). A healthy locked source with an open stream keeps its last frame on screen. Also add a same-LAN WebRTC rescue: synthesize a host candidate at the device's LAN IP (location.hostname) for each UDP port it advertises, so the browser probes the device directly instead of relying on NAT hairpinning of the STUN-reflexive candidate, which most home routers refuse. Gated diagnostics behind ?webrtclog=1.
Add ESP_LOGI in the esp_peer on_msg path so the serial log shows every local candidate the agent advertises (host vs srflx) and when the answer SDP is ready. Needed to confirm whether esp_peer enumerates a host candidate for the LAN IP or only offers the STUN-reflexive one.
esp_peer embeds only its STUN-reflexive (public) candidate in the answer SDP and never enumerates a host candidate for the device's LAN IP. When the browser is on the same LAN behind the same NAT, reaching that public address requires router hairpinning, which most home routers refuse, so ICE loops on unanswered binding requests and the agent disconnects after ~10 s. The serial log confirmed a single local candidate (local0:<public>:port) and no "local candidate N" trickle, i.e. the candidate is inline in the SDP, which is also why the browser-side synth from ans.candidates had nothing to work with. Parse the local UDP port from the answer SDP (rport when present, else the port-preserved srflx port) and synthesize a host candidate at each up interface IP (WiFi/Ethernet), returned to the browser alongside the answer. The browser then probes the device directly on the LAN instead of hairpinning. Logs each parsed answer candidate and each injected host candidate for verification.
On a network where ICE can't pair (esp_peer probes only from its reflexive candidate and can't resolve the browser's mDNS host candidates), the browser's auto-retry re-ran the negotiation on every page load and every 8 s. Each negotiation opens ICE/DTLS sockets on the device, whose lwIP pool is small; over ~20 min of testing they piled up until httpd's accept() failed with ENFILE (errno 23) and the HTTP server wedged - the whole device, not just WebRTC. Make WebRTC single-shot and non-churning: - Browser: one attempt per page; on failure record it in sessionStorage so reloads don't restart the storm. Stay on MJPEG, which is reliable. ?webrtcforce=1 clears the sticky flag to retry after a network change. - Device: drop the 1.8 s candidate-settle wait. esp_peer embeds both the host and srflx candidates in the answer SDP by the time it is ready, so there is nothing to wait for; blocking only held the HTTP worker and its socket longer, feeding the same ENFILE exhaustion.
WebRTC couldn't pair on same-LAN/NAT networks (esp_peer hairpins its reflexive candidate; the router refuses it) and never worked off-LAN at all. A TURN relay fixes both and enables remote access. Use coturn's use-auth-secret mode (RFC 5766 REST): the device stores only a shared secret (NVS keys turn_url, turn_secret) and derives short-lived HMAC-SHA1 credentials on demand (turn_cred.c, via mbedtls), so no long-term password is exposed. The same secret authenticates the device's esp_peer agent and, through GET /webrtc/ice, the browser - both relay through the same server. - turn_cred: username = expiry timestamp, password = base64(HMAC-SHA1(secret, username)); requires SNTP-set clock. - webrtc_kvm: add the TURN entry to the esp_peer ICE server list when configured; new webrtc_kvm_ice_config_json() builds the browser's iceServers (STUN + fresh TURN creds). - http_server: GET /webrtc/ice (auth-gated) serves that JSON; /config GET/POST carry turn_url + turn_secret (secret write-only, exposed only as turn_secret_set). max_uri_handlers 20 -> 21.
Before creating the RTCPeerConnection, GET /webrtc/ice and use the returned iceServers (public STUN plus a TURN relay with fresh short-lived credentials when the device has one configured) so candidate gathering includes relay candidates. Falls back to public STUN alone if the endpoint is absent or fails. Add TURN relay URL + secret fields to the Setup panel, following the existing write-only-secret convention (turn_secret sent only when typed; state shown via turn_secret_set).
Step-by-step coturn use-auth-secret configuration for the public host, firewall ports, device Setup fields, verification, and the peer-IP deny list that keeps the relay from being abused as an open proxy.
The device already runs on a WireGuard tunnel - an encrypted (ChaCha20), flat, directly-routable network with no NAT, hairpin, station isolation or mDNS. A browser that joins the same tunnel reaches the device at its tunnel IP directly, so WebRTC pairs over it with none of the problems that break same-LAN P2P and without any public TURN relay. Since the README already mandates a VPN for access, this is the natural transport. Advertise the configured WG local IP as an ICE host candidate (same local port; esp_peer binds on all interfaces). Harmless for a non-tunnel browser - its check to the tunnel address just fails and ICE falls back to LAN/srflx/TURN. Gated on P4KVM_WG_ENABLE.
Owner
|
This is really cool! I will try to find some time to test this. I did have a separate repo using WebRTC when I was still using the H264 encoder so I could dig that out if it's useful. |
Author
|
Please check again. It works, but I am having some stride/color ordering problems. Maybe you can figure it out? |
Owner
|
Yea, I had those exact same issues, which is why I switched to RGB, it's the only way I could get it to work. I think no modules on the esp32-p4 can deal with real YUV data 😛 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I took the POC and ran with it for a bench session on a rev 1.3 ESP32-P4-nano. This grew into a larger branch than I originally planned - happy to split it into smaller PRs if you prefer. Everything below was developed and tested against ESP-IDF 6.0 with the device on WiFi, using a new built-in test-pattern source (no TC358743 attached on my bench yet), so real HDMI capture still needs testing on your setup - that is the main thing I would love your help verifying.
Highlights
Video
P4KVM_TEST_PATTERN): animated color bars generated in the native pixel format, published through the same ring/semaphore contract as the CSI DMA - the whole downstream stack (JPEG, HTTP, UI, HID) runs on a bare P4 board. This is how most of the branch was hardware-verified.Input
Web UI
/statsJSON endpoint (capture/encode fps and times, JPEG size, HDMI state, recoveries, heap, IPs, and so on).Networking
ws_pre_handshake_cbbecause esp_http_server completes the 101 upgrade before the URI handler runs (an in-handler check would be too late).Fixes for issues in the README
Verification status (honest)
Hardware-verified on a rev 1.3 P4-nano over WiFi with the test pattern: boot, WiFi/mDNS/HTTP, /stats, MJPEG streaming at a steady 33 fps, YUV422 encoder byte order (pixel-checked bars), both video modes, the new UI. Host-side test for the stream parser. NOT yet verified: real HDMI capture through the TC358743, HID against a live host, the recovery ladder under real conditions, ATX outputs, Ethernet, auth-enabled operation, the WireGuard tunnel against a live peer, audio.
CHANGELOG.md has the full list; README documents wiring, trade-offs and the verification status. All measured numbers in the docs come from the actual board, not estimates.
Update: USB virtual media, serial console, and the HTTP wedge
A second bench session added two more USB functions and hunted down a socket bug. All built and pushed to this branch; still test-pattern/WiFi only, no target attached yet.
USB virtual media (MSC)
.img/.isoup to 12 MiB (the PSRAM cap, shared with the video buffers) streams straight from the browser into PSRAM and mounts read-only by default. A SCSI UNIT ATTENTION on remount makes the host re-read the new capacity. Endpoints:GET /media/status,POST /media/image,POST /media/eject; a top-bar disc icon + popover in the UI drives it. On hardware I watched a 2 MB image stage and mount end-to-end.tud_msc_*callbacks marked weak so the app's raw-block backend overrides them (the managed component only ships FAT-on-flash/SD). Documented incomponents/espressif__esp_tinyusb/PATCHES.md.USB serial console (CDC) + in-browser terminal
/dev/ttyACM0is exposed to the target as a third USB function, bridged to an xterm.js terminal over a/serialWebSocket (a KVM/TERM toggle in the top bar). Serial-over-LAN for headless boxes, kernel panics and BIOS/U-Boot serial menus. Guarded byCONFIG_TINYUSB_CDC_ENABLED, so disabling it falls back cleanly to HID+MSC. On hardware the composite enumerates (VID 0x303a/PID 0x4004) and/ws+/serialboth complete the WebSocket upgrade. README has a Linux-target section (ModemManager udev-ignore rule,serial-getty,console=ttyACM0).Fix: the "HTTP wedge" (accept ENFILE) turned out to be two bugs
max_uri_handlerswas below the number of registered routes, so esp_http_server silently dropped/wsand/serial("no slots left"); the browser then retry-stormed the missing/wsand exhausted the socket pool. Raised the cap with headroom.CONFIG_LWIP_TCP_MSLto 5 s and raisedCONFIG_LWIP_MAX_ACTIVE_TCPto 24.Verification for this update: composite USB enumeration, MSC image mount,
/wsand/serialupgrades, and the handler-registration fix are confirmed on the rev 1.3 board; the TIME_WAIT fix is flashed and pending a longer soak. Not yet verified end-to-end against a live target: booting from a mounted image, and CDC console traffic with a machine attached.