IP-KVM: video pipeline, USB HID/MSC/CDC, WireGuard, web console - #1
Closed
DatanoiseTV wants to merge 21 commits into
Closed
IP-KVM: video pipeline, USB HID/MSC/CDC, WireGuard, web console#1DatanoiseTV wants to merge 21 commits into
DatanoiseTV wants to merge 21 commits into
Conversation
Video: Kconfig-selectable capture pipeline. RGB888 (default, JPEG 4:2:0) or YUV422 via TC358743 UYVY output with a BitScrambler DMA loopback pass rotating each 32-bit word to the YVYU order the P4 JPEG encoder requires (the CSI-bridge color converter needs rev >= 3.0 silicon; this target is rev 1.x). All format-dependent constants derive from one descriptor block in capture_priv.h. The per-frame 6 MB cache msync in the encode loop is replaced by a one-time writeback+invalidate after allocation: the ring is DMA-only and the JPEG/BitScrambler drivers sync their buffers per run (invariant documented at the ctx definition). Test pattern (P4KVM_TEST_PATTERN): CPU-generated animated color bars in the native pixel format, published through the same ring/semaphore contract, so the whole downstream stack runs without the HDMI-CSI module. HID: add an absolute-pointer report (ID 3, 0..32767 axes) so tablet mode positions the host cursor 1:1 without drift; drop the abs-to-rel emulation. Relative mouse and boot keyboard unchanged. ATX: optional power/reset front-panel control via optocoupler GPIOs; 300 ms tap or 5 s force-off hold, per-pin busy guard, POST /atx. HTTP: TCP_NODELAY on stream and input sockets (multipart trailer chunks no longer wait on delayed ACKs), /stats JSON with per-stage pipeline timings, optional Basic auth on all routes - /ws gated via ws_pre_handshake_cb because esp_http_server completes the 101 upgrade before the URI handler runs. Recovery: skip HPD cycling while DDC5V is absent (slow 60 s retry in case the bit reads low transiently), escalate to a full TC358743 register re-init after three failed hotplug recoveries. All four configurations (default, yuv422, auth+atx+yuv422, testpat) build against ESP-IDF 6.0. Not yet validated on hardware; see README verification status.
The ESP32-P4 has no radio; boards like the P4-nano carry an ESP32-C6 over SDIO. esp_wifi_remote + esp_hosted proxy the standard esp_wifi API to it, so the STA code is plain esp_wifi with auto-reconnect. Modem power save is disabled: a KVM cares about input latency more than milliwatts. mDNS and netif/event-loop setup move out of ethernet.c into net_common.c so they run for either interface; Ethernet and WiFi can be up simultaneously. Off by default (P4KVM_WIFI_ENABLE); SSID/password in menuconfig, SDIO pinning in esp_hosted's own Kconfig section. Builds verified for the default (WiFi off) and sdkconfig.ci.wifi configurations.
Tablet (absolute pointer) becomes the default input mode: the host cursor tracks the browser pointer 1:1 with no pointer lock; clicking the video captures the keyboard (Esc is forwarded since there is no lock to escape). Relative pointer-lock mode stays for BIOS/UEFI and games that ignore tablet devices. The MJPEG client is rewritten around an incremental multipart parser that writes each JPEG byte exactly once into its final buffer (the old parser re-scanned and re-copied the whole accumulation buffer on every network chunk) and a decode loop that drops stale frames instead of queueing them. Parser covered by a host-side test (1-byte chunk splits, malformed-part resync, binary bodies containing CRLFCRLF). Adds: Ctrl+Alt+Del button, fullscreen, stats overlay (client draw fps + link Mbps + device /stats polled while visible), ATX power/reset/force- off buttons with confirmations (shown only when the firmware reports the GPIOs wired, probed with retries so a slow boot cannot hide them), staggered startup requests to avoid the lwIP connect burst, numpad and PrintScreen/ScrollLock/Pause key mapping.
README rewritten around the new feature set with explicit hardware constraints (rev 1.x silicon limits, 100 Mbit/s Ethernet ceiling, 2-lane MIPI budget, WiFi bitrate expectations) and an honest verification-status section: what is build- and host-test-verified versus what still needs on-hardware validation. CHANGELOG started at 0.2.0 (Keep-a-Changelog format). Ignore build_*/ variant directories.
IDF 6.0 defaults the P4 to rev >= 3.01, and rev <3 / >=3 images are mutually incompatible silicon targets. On the rev 1.3 board a rev-3-targeted image boot-loops with an immediate illegal-instruction panic before any bootloader output. The README already required selecting rev <3 in menuconfig; encode it in the defaults so a fresh idf.py set-target cannot silently produce a non-booting image.
New from-scratch shell: broadcast-monitor aesthetic (graphite, single amber signal accent, monospace telemetry, hairline panels), a proper no-signal state - animated static, decoded reason (DDC 5V absent / no TMDS / no sync / stream offline / device unreachable) and a fix hint - plus amber crop-mark brackets while the keyboard is captured. Right-hand drawer replaces the settings dropdown: keyboard helpers, host power, live JPEG-quality slider, pointer sensitivity, and a diagnostics table (stream/input/USB HID state, decoded HDMI source, pipeline, capture/encode fps and times, JPEG size, link rate, viewers, recoveries, encoder errors, heap/PSRAM, uptime, IP addresses, version). /stats gains usb_hid, ip_eth/ip_wifi, hostname and version fields for the panel. tc358743_sys_status now propagates I2C errors instead of returning garbage when the bridge is absent - the module-missing case now reads as 'device unreachable/no source' rather than random status bits. The proven engines (multipart parser, frame-dropping painter, HID queues, WS lifecycle, startup stagger) carry over unchanged; parser host tests still pass. Bundle: 32 KB single file.
First hardware run showed every reorder pass writing 4147196 of 4147200 bytes: with 64-bit prefetch and 32-bit consumption the downstream trailing counter starts before the final word drains, so trailing_bytes 8 left one 32-bit word in the pipe and the strict length check then discarded every frame (stream stayed black). Budget 16 trailing bytes - surplus zeros fall past the output DMA descriptor list and are clipped. Also throttle the failure log (it spammed the UART at frame rate) and count drops in enc_errors so /stats surfaces them.
Second hardware finding: with trailing_bytes 16 all 4147200 frame bytes arrive but the run returns ESP_ERR_TIMEOUT - the surplus trailing zeros have no output descriptor to land in, so the engine never raises EOF. The pipeline evidently holds 96 bits in flight (64-bit prefetch plus one 32-bit read stage), not 64. Pad the output buffer by 64 bytes and accept written >= frame_bytes; the encoder still reads exactly frame_bytes.
Hardware verification with the color-bar test pattern settled it: every bar decoded as if the encoder reads chroma-first, meaning IDF's JPEG_ENCODE_IN_FORMAT_YUV422 == FOURCC 'YVYU' names the little-endian 32-bit WORD value, not the byte sequence. Byte-wise the encoder wants U Y V Y - exactly the TC358743's native UYVY stream. The reorder pass was built on the opposite reading of jpeg_types.h and produced swapped chroma; it also measured ~28 MB/s (147 ms per 1080p frame, capping the pipeline at 5.6 fps), so it was doubly untenable. The YUV422 pipeline now feeds the framebuffer straight to the encoder (measured encode: 29 ms/frame at 1080p, 360 MHz). Kconfig symbol renamed P4KVM_PIPELINE_YUV422_BS -> P4KVM_PIPELINE_YUV422; the bsasm program stays in-tree as reference with the pipeline-depth findings documented (96 bits in flight, trailing-flush behavior).
WireGuard (P4KVM_WG_ENABLE): vendored trombik/esp_wireguard 0.9.0 into components/esp_wireguard with three IDF 6 patches - mbedtls 4 removed mbedtls/entropy.h, so the RNG bootstrap (CTR-DRBG layered on an entropy source that itself wrapped esp_fill_random) now draws from the hardware TRNG directly; two GCC 15 -Werror suppressions for a stringop-overread false positive in x25519 and the deliberately unterminated WireGuard protocol constants. The supervisor task waits for an IP, syncs NTP time (handshakes need wall-clock), connects, and re-handshakes after 30 s of stale peer. Keys/endpoint via menuconfig; tunnel state exposed as 'wg' in /stats and a WIREGUARD row in the diagnostics panel. WiFi reliability: reconnect moved off the event-loop task (the handler previously slept 1 s inside the loop) onto a one-shot esp_timer with exponential backoff 0.5-8 s, reset on every got-IP; all-channel scan selecting the strongest BSS (repeater/mesh setups); in-supplicant failure_retry_cnt=3; power save re-asserted WIFI_PS_NONE after every association since some coprocessor firmwares reset it on reconnect. Builds verified for the default (WG off) and sdkconfig.ci.wg configs.
Capture resolution becomes a boot-time mode persisted in NVS instead of a compile-time constant. Each mode carries its own EDID so the HDMI source actually outputs that timing: the 720p60 EDID is generated from the 1080p30 template (canonical CEA VIC-4 DTD, native SVD, checksums recomputed and parse-verified programmatically). Switching - drawer Video section or POST /video-mode - restarts the device, which a resolution change requires anyway (new EDID negotiation, re-sized DMA buffers). 720p60 is the new default: ~1/3 the MJPEG bitrate of 1080p at equal quality, matching what a 2.4 GHz WiFi link sustains (the C6 radio tops out around 20-40 Mb/s TCP; 1080p q70 at ~60-70 Mb/s is why fps sawtooths on WiFi while the encoder holds a measured steady 33 fps). HID absolute coordinates are now resolution-independent: the browser scales to the 0..32767 HID logical range itself and the firmware passes them through, so pointer mapping never depends on the active mode. The web canvas tracks the device-reported frame size from /stats.
Same approach as h2c-rpi: the TC358743 already emits the HDMI source's audio on its I2S pads (this firmware configures that path; the bridge is I2S master with clocks derived from the stream). Three jumper wires bring BCK/LRCK/DATA to P4 GPIOs set in menuconfig; the P4 receives as I2S slave (32-bit slots, top 16 bits kept) and streams raw PCM S16LE 48 kHz stereo over the /audio WebSocket in 10 ms frames (~1.5 Mbit/s), single listener, auth-gated like /ws. The UI gains a speaker toggle (shown only when the firmware reports audio wired) feeding an AudioWorklet ring buffer (~250 ms, silence on underflow), plus an AUDIO diagnostics row and an 'audio' field in /stats. Off by default; 48 kHz is assumed (the only rate HDMI mandates and the only one the EDID advertises). Compile-verified for enabled and disabled configs; hardware-unverified until the I2S pads are wired.
The stock config assumed 2 MB flash (bootloader warned every boot) and the single-app layout capped the factory partition at 1 MB - the app was at 95 % of it. Custom partition table with a 4 MB factory slot on the P4-nano's 16 MB flash, leaving room to grow and for a future OTA split. Also provide an explicit TinyUSB device descriptor (Espressif VID, generic TinyUSB PID, our existing strings) instead of the NULL that made the stack log fallback warnings at boot.
New runtime_cfg module: NVS-backed settings with Kconfig fallbacks, served by GET/POST /config (cJSON; moved to the registry component in IDF 6). Secrets - WiFi password, WireGuard private and preshared key - are write-only: reads return only configured/not-configured flags. Consumers (wifi_net, wireguard_net, atx_ctrl) read the store once at init, so the UI saves and restarts via the shared deferred-restart path. The drawer gains a Setup section (WiFi credentials, WireGuard endpoint, port, keys, tunnel IP, ATX GPIOs and polarity) with save-and-restart, and the Host Power section is now always visible - when the GPIOs are not configured it explains how to wire and set them instead of hiding. WiFi and WireGuard support are now compiled in by default (activation is a runtime decision: empty SSID or missing keys simply skip startup), so a stock build can be fully configured from the browser.
A stale sdkconfig kept CONFIG_P4KVM_WG_ENABLE unset from before the Kconfig default flipped to y, so the build silently dropped WireGuard and the tunnel state was always "off" regardless of pushed config. Pin WIFI/ETH/WG on here so a pre-existing sdkconfig can never build a core feature out again.
Toggleable HUD (default on) pinned to the stage corner, showing the live pointer report (absolute wire coords + pixel position, or relative deltas, plus button/wheel state) and keyboard activity (held keys with modifiers and a rolling list of recent keystrokes). Reads straight from the HID send path so it reflects what actually goes on the wire, and updates even when the input link is down. The no-signal overlay is now pointer-transparent so the HUD tracks the pointer on the test screen too. Lets you debug input without a host or serial console.
esp_wireguard registers a raw lwIP netif and stores its own device struct in netif->state. With LWIP_ESP_NETIF_DATA=0 (the default), esp_netif also uses netif->state for its back-pointer, so its global ext-callback (esp_netif_internal_dhcpc_cb) reinterpreted the WireGuard device as an esp_netif_t and null-dereferenced ip_info the instant the tunnel address was set -> Load access fault boot loop right after 'allowed_ip'. lwipopts.h only moves esp_netif's pointer into netif->client_data (freeing netif->state for third parties) when the bridge or PPP option is on; enable the bridge purely for that side effect. Verified on hardware (rev 1.3): boots clean past allowed_ip and reaches 'peer up'.
Present a removable USB mass-storage LUN to the target alongside the HID keyboard/mouse, backed by a PSRAM ramdisk. Boots as an empty formatted FAT12 floppy; an arbitrary .img/.iso up to 12 MiB can be streamed in from the browser and mounted, so the target can boot rescue/installer images without physical media. - main/usb_msc.c/.h: dynamic-capacity ramdisk with begin/load/commit image flow. A deferred UNIT ATTENTION (0x28/0x00) on remount forces the host to re-read the new capacity. Bounds-checked read10/write10, eject via START_STOP_UNIT, write-protect toggle. - main/usb_hid.c: composite descriptor now advertises HID (itf 0) + MSC (itf 1). Separate FS (bulk 64) and HS (bulk 512) config descriptors as the high-speed spec requires; usb_msc_init() runs before the driver install. - main/http_server.c: GET /media/status, POST /media/image (raw body, ?writable=), POST /media/eject. The image streams from the socket straight into PSRAM in 4 KiB chunks with no full-image bounce buffer. - WebUI: Virtual Media panel with file picker, writable toggle, XHR upload progress, mount/eject and live status. - components/espressif__esp_tinyusb: vendored from managed 2.1.1 with the six tud_msc_* callbacks weakened so the app's strong raw-block definitions win (esp_tinyusb only ships FAT-on-flash/SD backends). See PATCHES.md. Config: CONFIG_TINYUSB_MSC_ENABLED, BUFSIZE 8192, pinned in sdkconfig.defaults.
Expose a virtual serial port to the target as a third USB function (HID+MSC+CDC) and bridge it to a browser terminal, turning the KVM into a serial-over-LAN console for headless boxes, kernel panics, and BIOS/U-Boot serial menus. - main/usb_serial.c/.h: CDC-ACM port via esp_tinyusb's CDC API. Target->browser bytes stage through an 8 KiB stream buffer drained by a pump task that sends WS frames; browser keystrokes are written straight to the target. Single terminal client, newest wins, dropped from the httpd close hook. Non-blocking throughout so a stalled browser never back-pressures the target's console. - main/usb_hid.c: composite descriptor gains the CDC IAD+comm+data interfaces (itf 2/3, EP 0x83 notif, 0x04/0x84 bulk), guarded by CONFIG_TINYUSB_CDC_ENABLED so it cleanly falls back to HID+MSC when disabled. FS/HS share one CDC fragment macro differing only in bulk max-packet size. - main/http_server.c: /serial WebSocket (auth pre-handshake like /ws), serial state added to /stats. - WebUI: KVM/TERM view toggle in the top bar; TERM shows an xterm.js terminal wired to /serial, connecting only while the terminal is open. Terminal dot mirrors the target's DTR from /stats. - README: virtual-media and serial-console sections, including a Linux target guide (ModemManager udev ignore rule by VID/PID, serial-getty and console=ttyACM0 setup) so the console works without breaking things. Both features build and link but are hardware-unverified (port busy this session); flagged as such in the README.
max_uri_handlers was 12 but 15 handlers register once MSC + serial were
added, so httpd silently dropped /ws and /serial ("no slots left"). The
browser's WebSocket reconnect storm against the missing /ws then exhausted
the lwIP socket pool (accept -> ENFILE, error 23) - the long-hunted "HTTP
wedge", now confirmed from the boot log. Raise to 18 with headroom, and
document the count so future handlers bump it.
Also remove the temporary socket-diagnostic task now that the cause is found.
WebUI: virtual media moves out of the settings drawer into a top-bar disc
icon + popover, so the mounted image and mount/eject are one click away
without opening settings. The icon glows amber while an image is mounted;
the popover shows live status and upload progress.
Even with all URI handlers registered, the browser re-fetches /stream (and reconnects /ws, /serial) on any hiccup, and lwIP's default 60s MSL held each closed socket in TIME_WAIT for a full minute. A few reconnects per minute then filled the 24-socket pool and httpd accept() returned ENFILE (errno 23) - the same wedge symptom, now from socket lifetime rather than handler slots. It took ~8 min to accumulate instead of ~30s. Drop CONFIG_LWIP_TCP_MSL 60000 -> 5000 (ample on a LAN/tunnel) so churned sockets free almost immediately, and raise CONFIG_LWIP_MAX_ACTIVE_TCP 16 -> 24 to match the socket pool and give the httpd sessions headroom.
Owner
Author
|
Duplicate; the upstream PR jrowny#1 already tracks this branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the ESP32-P4 + TC358743 IP-KVM from the initial import to a working
console: 1080p/720p MJPEG video, USB HID, virtual media, a serial console,
and remote access over WireGuard, all driven from a single-file web UI.
Video
on the 100 Mbit PHY) and YUV422/UYVY fed straight to the encoder. The YUV422
byte order is hardware-verified on rev 1.3 (the encoder consumes native UYVY;
an earlier BitScrambler reorder pass was removed after the color-bar test
disproved its premise).
applied with a restart since it re-advertises EDID and resizes DMA buffers.
HDMI module.
Input (USB HID)
0..32767, 1:1 cursor with no capture). Tablet and pointer-lock modes in the UI.
Virtual media (USB MSC)
floppy; an arbitrary .img/.iso up to 12 MiB streams from the browser into
PSRAM and mounts (read-only by default), with a SCSI UNIT ATTENTION on remount
so the host re-reads capacity.
GET /media/status,POST /media/image,POST /media/eject. Top-bar disc icon + popover in the UI.app's raw-block backend wins (see components/.../PATCHES.md).
Serial console (USB CDC)
in-browser xterm.js terminal over /serial (KVM/TERM toggle). README documents
the Linux target setup (ModemManager udev-ignore, serial-getty, console=).
HID+MSC.
Networking
(esp_wifi_remote + esp_hosted over SDIO).
fixes an esp_netif/raw-netif state collision that crash-looped the dhcpc
callback (LWIP_ESP_NETIF_DATA via the bridge option).
Web UI
keyboard/power helpers, a live debug overlay, settings, and the media popover.
HTTP robustness (this branch's last fixes)
silently dropped them, and the browser's reconnect storm against the missing
/ws exhausted the socket pool -> accept ENFILE).
stream/WS reconnects can no longer fill the 24-socket pool.
Verification status
Verified on a rev 1.3 board (WiFi, test pattern): boot, WiFi/Ethernet + mDNS,
HTTP + /stats + MJPEG, YUV422 byte order, WireGuard tunnel up, USB composite
enumeration, MSC image mount, /ws and /serial WebSocket upgrade, and the socket
fixes. Not yet validated against real HDMI capture through the TC358743, HID
against a live host, the recovery ladder, ATX outputs, or the CDC end-to-end
byte bridge with a target attached.