feat(compositor): CPU backend — WARP render + software decode, surfaced to the user - #162
Open
EtienneLescot wants to merge 206 commits into
Open
feat(compositor): CPU backend — WARP render + software decode, surfaced to the user#162EtienneLescot wants to merge 206 commits into
EtienneLescot wants to merge 206 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…s and snapping - feat(ai-edition): multi-session chat history + multi-clip timeline with drag-drop - feat(ai-edition): timeline interaction P0 — clip resize/delete/duplicate, region drag, Edit Clip dialog - feat(ai-edition): timeline viewport spine — pan/zoom/reorder (T01-T09) - feat(ai-edition): timeline lanes, navigator strip, Place-skip mode (T10-T18) - feat(ai-edition): timeline polish — scrub preview, resequence, snap guides (T19-T25)
…r pipeline into the preview
… and manual compaction
…nd move the transport into the timeline
…nment, then CTranslate2
…e thinking on/off
…atch the v4 design - feat(ai-edition): refactor editor to OpenScreen Editor v4 (Media/Edit/Rec) - style(ai-edition): flatten timeline region pills to v4 design - style(ai-edition): match v4 timeline navigator to design - style(ai-edition): align v4 timeline clip cards + ruler to design - fix(ai-edition): hide timeline lane watermark when lane has regions - style(ai-edition): float clip label as v4 blur chip over waveform - docs(ai-edition): add v4 design-parity + import audit notes
…ounded in the editor
…d add LLM effect tools
EtienneLescot
marked this pull request as ready for review
July 27, 2026 22:49
EtienneLescot
added a commit
that referenced
this pull request
Jul 28, 2026
Empty commit so the scoping PR can exist; drop it when the real work starts. Same move as #162 for the Windows CPU backend.
EtienneLescot
added a commit
that referenced
this pull request
Jul 28, 2026
Closes part of PR #183 (the scoping body). Defines the target architecture (wgpu backend mirroring the D3D11 seam #162 introduced), the per-platform frame contract, the WGSL shader port inventory (1:1 with shaders.hlsl's 9 entry points), the cosmic-text replacement for text.rs, the build/CI plumbing for Linux, and a WP0..WP7 milestone plan. Adds a Linux port subsection to native-compositor.md and a doc-table entry to README.md. The spec body of PR #183 is updated separately to reference this doc.
EtienneLescot
added a commit
that referenced
this pull request
Jul 28, 2026
…d skeleton Implements the foundation of PR #183. WP0 — Cargo target gating - workspace deps: wgpu 24 (cross-platform, Vulkan/Metal/D3D12; pinned to the windows-0.58-compatible line so the existing D3D11 path stays on its validated toolchain), pollster for block_on. - compositor/Cargo.toml: wgpu + pollster deps cross-platform, the windows dep kept for the D3D11 path. - compositor-view-napi: the windows dep stays; lib.rs is split — the existing napi surface is cfg(windows)-gated into windows.rs, a non-Windows placeholder stands in until WP7 ships the Linux napi skeleton. - compositor/src/lib.rs: cfg(windows) on d3d/cpu_frames/text/compositor/ live/pipeline (the D3D11 path); portable modules + new vk* stay ungated so the crate compiles everywhere. - poc-d3d: cfg(windows) on the Win32 body; non-Windows stub so `cargo check` on the target stays green. - compositor/build.rs: cross-build detection via cfg!(host_os) vs CARGO_CFG_TARGET_OS. When target != host, skip cc/bindgen and write a cached or stub ffi.rs so `cargo check --target x86_64-unknown-linux-gnu` from a Windows host passes without a cross-toolchain. Path separator made platform-aware. WP1 — vk.rs (wgpu device + probe + lavapipe classification) - VkGpu { device, queue, backend, adapter_info } mirrors d3d::Gpu shape. - create (hardware-strict, for tests/goldens), create_blocking, create_auto (fallback HW -> lavapipe; full semantic equivalence with d3d::Backend in WP1+). - probe cached via OnceLock; returns the same "none" semantics as a missing addon on a machine without a Vulkan implementation. - classify() detects lavapipe by adapter name on Linux (llvmpipe / lavapipe substrings) and WARP on Windows — both routed to BackendKind::Cpu so probeBackend() reports "cpu" exactly as the Windows WARP rung does today. - Unit tests: probe-doesn't-panic, classify-hardware, classify-lavapipe. WP2 — vk_frames.rs (NV12 upload skeleton) - VkFrameTex { y: r8unorm, uv: rg8unorm, width, height } mirrors the D3D11 NV12 texture that cpu_frames.rs presents today; the AVFrame carrier (data[0] = Box ptr) is the contract #162 introduced, generalised to carry the platform payload through crate boundaries. - VkFrames: AVFrame slot (av_frame_alloc), ensure_textures(w, h) creates the two R8/Rg8 textures when dimensions change. - Drop frees the AVFrame slot. - The swscale -> AV_PIX_FMT_NV12 -> queue.write_texture path lands in a follow-up WP2 commit; this commit establishes the seam and types. vk_render.rs — module skeleton with VERSION constant and a test. vk_shaders/blit.wgsl — minimal placeholder WGSL (fullscreen triangle + black clear) for the include_str! source of the slice renderer. Verified: - `cargo check --workspace` green - `cargo check --workspace --target x86_64-unknown-linux-gnu` green (uses OPENSCREEN_BINDGEN_CACHE_PATH to feed the bindgen output from a prior host build into the cross-build — see build.rs write_stub_ffi.) WP3+ (slice renderer with WGSL core shaders and cross-backend golden diff) is scoped in linux-compositor-port.md but lands in follow-up commits to keep this PR reviewable.
3 tasks
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 28, 2026
Branch off the CPU-backend seam established by PR #162 and split the compositor crate into per-platform directories. Windows behaviour is byte-identical (81 tests pass) and the macOS stubs that follow this commit in the stack expose the same public surface (Backend, Gpu, Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams, LayerCB...) and Err from every operation. Renames + cfg dispatch: - crates/compositor/src/d3d_windows.rs (from d3d.rs) - crates/compositor/src/cpu_frames_windows.rs (from cpu_frames.rs) - crates/compositor/src/compositor_windows.rs (from compositor.rs) - crates/compositor/src/pipeline_windows.rs (from pipeline.rs) - crates/compositor/src/text_windows.rs (from text.rs) - crates/compositor/wrapper_windows.h (from wrapper.h) lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across the per-platform modules so call-sites stay portable. live.rs: Win32 harness (run_standalone, host_proc, wide, client_size) moved into a #[cfg(windows)] pub mod standalone_harness with a #[cfg(target_os = "macos")] no-op stub for run_standalone (poc-d3d is dev-only on Windows). Cargo.toml: macOS deps (metal, objc, block, core-foundation) added under [target.cfg(target_os = macos).dependencies]; windows dep gated to cfg(windows). compositor-view-napi mirrors the gate. crates/.cargo/config.toml: target-specific env block for macOS (LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs). build.rs: selects wrapper_windows.h vs wrapper_macos.h on CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin + -isysroot on macOS, finds MAC_FFMPEG_DIR or vendored thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared. Stacked on feat/d3d-warp-fallback (PR #162).
3 tasks
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 29, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into a failing test, and says plainly what the rest is worth. The two backends cannot run on the same machine, so iso cannot be established by comparing two rendered images. What can be — and what is the layer where divergence actually happened on this branch — is the geometry: `plan_frame` is literally the same code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit` pins them for a scene that exercises padding, crop, a displaced PiP camera, corner radii and the zoom, comparing with `to_bits()` rather than an epsilon. It runs in BOTH CI jobs — `rust-macos-compositor-check` and `rust-windows-compositor-check`. If the two platforms ever compute different placements, one of them goes red. That is the guarantee that was being claimed and never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely because nothing compared them. What this test does NOT cover, and must not be read as covering: rasterisation. D3D11 and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical with a max deviation of 3/255 between two backends on the SAME machine, and that is the floor, not the target. Shader parity is held separately, by `shaders.metal` and `shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences came out of that pass, three of them live. The expected values are measured from this code, not chosen. A drift is a divergence to explain, not a tolerance to loosen.
EtienneLescot
added a commit
that referenced
this pull request
Jul 30, 2026
PR #183 rebased from #162 base (old branch) to release/v1.8.0 (the target the user asked for after conflict resolution on #162). The 41 extra commits between #162 and release/v1.8.0 are NOT in this PR — they represent upstream work (native GIF export pipeline, pixi.js removal, zoom/cursor fixes, etc.). The port reuses the same architecture and files (vk.rs, vk_decode.rs, vk_frames.rs, vk_render.rs, text_cosmic.rs, backend/{mod.rs,d3d11.rs,wgpu_backend.rs}, tests/vk_cross_golden.rs, live_lin.rs, pipeline_lin.rs, compositor-view-napi/src/linux.rs) but on top of the new base. Key adaptations vs the old base: - `compositor`/`d3d`/`live`/`pipeline`/`text`/`gif_export` are now `#[cfg(windows)]` modules in `compositor/src/lib.rs`. The Linux sibling modules (`vk*`, `backend`, `live_lin`, `pipeline_lin`, `text_cosmic`) are platform-agnostic (cross-compile via wgpu). - `compositor::LayerCB` moved to `backend::LayerCB` (the platform-neutral definition matches HLSL `cbuffer Layer` and WGSL `struct Layer` byte-for-byte). Compositor's `LayerCB` is now `use crate::backend::LayerCB`. - `compositor/src/compositor.rs` uses `use crate::backend::LayerCB` instead of defining its own. This unifies the two `LayerCB` symbols that previously caused a `E0308` mismatched-types error in d3d11.rs. - `pipeline` (D3D11 arm) is NOT renamed to `pipeline_win`. The D3D11 `run_composited`/`run_composited_multi` keep their original names and signatures. The Linux path (`pipeline_lin::Decoder`/`VideoEncoder`) is parallel. - `compositor-view-napi`: split into `lib.rs` (cfg-gated `mod windows` / `mod linux` + `pub use`) plus `windows.rs` (full Windows addon, 622 lines) plus `linux.rs` (slice-based Linux addon, 230 lines). This replaces the previous "stub + Windows content in lib.rs" pattern the old branch had. The Windows dependency is `target.cfg(windows)` gated, so Linux builds don't pull `windows = "0.58"`. - `compositor/build.rs` cross-build-aware: detects `cfg!(windows)` vs `CARGO_CFG_TARGET_OS` and either runs the full bindgen path (native host) or writes a cached/stub `ffi.rs` (cross-build, via `OPENSCREEN_BINDGEN_CACHE_PATH`). The wrapper.h includes `<libswscale/swscale.h>` (needed for `vk_frames.rs` swscale calls). - `compositor/wrapper.h` includes `<libswscale/swscale.h>` for swscale bindings used by `vk_frames.rs`. The build.rs allowlists `sws_.*`. - `cosmic-text = "0.19"` added as a workspace dep + compositor dep (the v0.19 features are default — there's no `fontdb`/`swash` feature flag in v0.19). - `poc-d3d`: bench.rs's `export_gif` call updated to the new signature (`&[ClipSource]` instead of `&screen, &webcam, Some(&cursor)`). `gpu`/`comp` are now hoisted early so the GIF shortcut can run before the rest of the MP4-only setup. `mod app;` / `mod bench;` in `poc-d3d/src/main.rs` and the `windows` dep in its `Cargo.toml` are now `#[cfg(windows)]`/target-gated so Linux builds skip them. What's in this branch (new files only): - `crates/compositor/src/vk.rs` — wgpu device + lavapipe probe (`BackendKind::Cpu` when adapter is software via lavapipe on Mesa). - `crates/compositor/src/vk_decode.rs` — `SwDecoder` (software ffmpeg decode via `avcodec`). - `crates/compositor/src/vk_frames.rs` — `VkFrames` (NV12 split upload to two wgpu textures, AVFrame carrier). - `crates/compositor/src/vk_render.rs` — `render_slice_c1` (slice renderer: WGSL layer.wgsl pipeline + bind groups + readback to PPM). - `crates/compositor/src/vk_shaders/{layer,blur}.wgsl` — WGSL ports of the HLSL `ps_main` mode 0/1/2 + Kawase chain. - `crates/compositor/src/text_cosmic.rs` — cosmic-text-based Linux text rasterizer (rustybuzz + swash + fontdb). - `crates/compositor/src/backend/{mod,d3d11,wgpu_backend}.rs` — `CompBackend` trait + D3d11Backend forwarder (wraps existing Compositor) + WgpuBackend Linux impl. - `crates/compositor/src/live_lin.rs` — Linux LiveView skeleton (cfg(not(windows))). - `crates/compositor/src/pipeline_lin.rs` — Linux pipeline arm (Decoder / VideoEncoder / ExportCodec / run_composited with VAAPI candidates: `h264_vaapi` + `libopenh264`). - `crates/compositor/tests/vk_cross_golden.rs` — slice integration test (decode frame 180 of the fixture, render, write PPM, log stats). - `crates/compositor-view-napi/src/{linux.rs,windows.rs}` — split napi addon arms per platform. Verified: - `cargo check --workspace` (Windows host) — green. - `cargo check --workspace --target x86_64-unknown-linux-gnu` (via `OPENSCREEN_BINDGEN_CACHE_PATH`) — green. - `cargo test --tests -p openscreen-compositor` — 106 tests pass (100 unit + 2 export_timing + 3 output_geometry_golden + 1 vk_cross_golden slice). The `vk_cross_golden` slice produces a 1920×1080 RGBA8 PPM with mean RGB ≈ 232 on this Windows AMD iGPU. What this branch deliberately does NOT include (WP4+–WP7 follow-ups per `linux-compositor-port.md` §7): - Full effect set on `WgpuBackend` (modes 4/6/7/8/9/10/11/12/13 — only modes 0/1/2 and blur are ported from HLSL today). - `compose_frame` orchestration port to trait dispatch (the 800-line pure Rust math in `compositor.rs::compose_frame` still runs through the D3D11 path). - Full Linux LiveView (multi-frame render thread) — `live_lin.rs` is a skeleton today. - Linux MP4 muxer full implementation (ffmpeg avformat_new_stream + write_header + write_trailer) — `pipeline_lin::run_composited` is a skeleton that opens the encoder + runs the 1-frame path; the full muxer lands in WP7. The architecture (`CompBackend` trait + `LayerCB` + cross-platform `vk_*` modules + `vk_cross_golden` end-to-end test) is in place to absorb each of these as small focused follow-up commits.
EtienneLescot
added a commit
that referenced
this pull request
Jul 30, 2026
…ing back to WARP PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP cannot serve this pipeline at all -- and not for the expected reason (speed): - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns DXGI_ERROR_UNSUPPORTED (0x887A0004). - Drop the flag and WARP creates at FL 11_1, but QueryInterface for ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles. pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so preview and export decode every frame on it. A WARP device would produce none: the retry would only convert a clear startup failure into an obscure ffmpeg one. crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and fails if Windows ever changes them. So there is no fallback, and the answer to "WARP for export or preview only?" is neither. What the compositor does instead is fail in a way the user can act on: - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT) to tell "this adapter has no video decoder" -- the RDP / VM case -- apart from "no FL 11_1 adapter at all", and says which, plus what to do. Export gets this for free: ExportDialog already renders the native message. - That message now reaches the preview too. create_view returns an id long before the render thread can die, so the failure existed only as an eprintln! and the user just saw a black canvas. The thread stores its fatal error in live::Shared, read_frame relays it as an Err on the next pull (~33 ms), and NativeCompositorOverlay renders it in place of the canvas. The WGC capture helper stays hardware-only on purpose: it requests no VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host whose compositor cannot start only produces footage the user can neither edit nor export.
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 30, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF refactor that extracted `walk_composited_timeline` for the GIF path. The conflict resolution put the WARP branching inside the timeline walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those are MP4-specific. This commit moves the encoder creation back into `run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope) and adds the `if software_frames { ... } else { ... }` branching in the per-frame `on_frame` closure. Also fixes the unclosed `run_gif_bench` delimiter from the conflict resolution and adapts the call site to `export_gif`'s new signature (`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF refactor moved from `(screen, webcam, cursor, out_path, params, dither)` to the slice-2 MP4-shaped signature).
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 30, 2026
Branch off the CPU-backend seam established by PR #162 and split the compositor crate into per-platform directories. Windows behaviour is byte-identical (81 tests pass) and the macOS stubs that follow this commit in the stack expose the same public surface (Backend, Gpu, Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams, LayerCB...) and Err from every operation. Renames + cfg dispatch: - crates/compositor/src/d3d_windows.rs (from d3d.rs) - crates/compositor/src/cpu_frames_windows.rs (from cpu_frames.rs) - crates/compositor/src/compositor_windows.rs (from compositor.rs) - crates/compositor/src/pipeline_windows.rs (from pipeline.rs) - crates/compositor/src/text_windows.rs (from text.rs) - crates/compositor/wrapper_windows.h (from wrapper.h) lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across the per-platform modules so call-sites stay portable. live.rs: Win32 harness (run_standalone, host_proc, wide, client_size) moved into a #[cfg(windows)] pub mod standalone_harness with a dev-only on Windows). Cargo.toml: macOS deps (metal, objc, block, core-foundation) added under [target.cfg(target_os = macos).dependencies]; windows dep gated to cfg(windows). compositor-view-napi mirrors the gate. crates/.cargo/config.toml: target-specific env block for macOS (LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs). build.rs: selects wrapper_windows.h vs wrapper_macos.h on CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin + -isysroot on macOS, finds MAC_FFMPEG_DIR or vendored thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared. Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 30, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into a failing test, and says plainly what the rest is worth. The two backends cannot run on the same machine, so iso cannot be established by comparing two rendered images. What can be — and what is the layer where divergence actually happened on this branch — is the geometry: `plan_frame` is literally the same code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit` pins them for a scene that exercises padding, crop, a displaced PiP camera, corner radii and the zoom, comparing with `to_bits()` rather than an epsilon. It runs in BOTH CI jobs — `rust-macos-compositor-check` and `rust-windows-compositor-check`. If the two platforms ever compute different placements, one of them goes red. That is the guarantee that was being claimed and never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely because nothing compared them. What this test does NOT cover, and must not be read as covering: rasterisation. D3D11 and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical with a max deviation of 3/255 between two backends on the SAME machine, and that is the floor, not the target. Shader parity is held separately, by `shaders.metal` and `shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences came out of that pass, three of them live. The expected values are measured from this code, not chosen. A drift is a divergence to explain, not a tolerance to loosen.
EtienneLescot
added a commit
that referenced
this pull request
Jul 30, 2026
…ing back to WARP PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP cannot serve this pipeline at all -- and not for the expected reason (speed): - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns DXGI_ERROR_UNSUPPORTED (0x887A0004). - Drop the flag and WARP creates at FL 11_1, but QueryInterface for ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles. pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so preview and export decode every frame on it. A WARP device would produce none: the retry would only convert a clear startup failure into an obscure ffmpeg one. crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and fails if Windows ever changes them. So there is no fallback, and the answer to "WARP for export or preview only?" is neither. What the compositor does instead is fail in a way the user can act on: - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT) to tell "this adapter has no video decoder" -- the RDP / VM case -- apart from "no FL 11_1 adapter at all", and says which, plus what to do. Export gets this for free: ExportDialog already renders the native message. - That message now reaches the preview too. create_view returns an id long before the render thread can die, so the failure existed only as an eprintln! and the user just saw a black canvas. The thread stores its fatal error in live::Shared, read_frame relays it as an Err on the next pull (~33 ms), and NativeCompositorOverlay renders it in place of the canvas. The WGC capture helper stays hardware-only on purpose: it requests no VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host whose compositor cannot start only produces footage the user can neither edit nor export.
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 30, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF refactor that extracted `walk_composited_timeline` for the GIF path. The conflict resolution put the WARP branching inside the timeline walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those are MP4-specific. This commit moves the encoder creation back into `run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope) and adds the `if software_frames { ... } else { ... }` branching in the per-frame `on_frame` closure. Also fixes the unclosed `run_gif_bench` delimiter from the conflict resolution and adapts the call site to `export_gif`'s new signature (`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF refactor moved from `(screen, webcam, cursor, out_path, params, dither)` to the slice-2 MP4-shaped signature).
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 30, 2026
Branch off the CPU-backend seam established by PR #162 and split the compositor crate into per-platform directories. Windows behaviour is byte-identical (81 tests pass) and the macOS stubs that follow this commit in the stack expose the same public surface (Backend, Gpu, Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams, LayerCB...) and Err from every operation. Renames + cfg dispatch: - crates/compositor/src/d3d_windows.rs (from d3d.rs) - crates/compositor/src/cpu_frames_windows.rs (from cpu_frames.rs) - crates/compositor/src/compositor_windows.rs (from compositor.rs) - crates/compositor/src/pipeline_windows.rs (from pipeline.rs) - crates/compositor/src/text_windows.rs (from text.rs) - crates/compositor/wrapper_windows.h (from wrapper.h) lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across the per-platform modules so call-sites stay portable. live.rs: Win32 harness (run_standalone, host_proc, wide, client_size) moved into a #[cfg(windows)] pub mod standalone_harness with a dev-only on Windows). Cargo.toml: macOS deps (metal, objc, block, core-foundation) added under [target.cfg(target_os = macos).dependencies]; windows dep gated to cfg(windows). compositor-view-napi mirrors the gate. crates/.cargo/config.toml: target-specific env block for macOS (LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs). build.rs: selects wrapper_windows.h vs wrapper_macos.h on CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin + -isysroot on macOS, finds MAC_FFMPEG_DIR or vendored thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared. Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot
pushed a commit
that referenced
this pull request
Jul 30, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into a failing test, and says plainly what the rest is worth. The two backends cannot run on the same machine, so iso cannot be established by comparing two rendered images. What can be — and what is the layer where divergence actually happened on this branch — is the geometry: `plan_frame` is literally the same code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit` pins them for a scene that exercises padding, crop, a displaced PiP camera, corner radii and the zoom, comparing with `to_bits()` rather than an epsilon. It runs in BOTH CI jobs — `rust-macos-compositor-check` and `rust-windows-compositor-check`. If the two platforms ever compute different placements, one of them goes red. That is the guarantee that was being claimed and never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely because nothing compared them. What this test does NOT cover, and must not be read as covering: rasterisation. D3D11 and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical with a max deviation of 3/255 between two backends on the SAME machine, and that is the floor, not the target. Shader parity is held separately, by `shaders.metal` and `shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences came out of that pass, three of them live. The expected values are measured from this code, not chosen. A drift is a divergence to explain, not a tolerance to loosen.
EtienneLescot
added a commit
that referenced
this pull request
Jul 31, 2026
…ing back to WARP PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP cannot serve this pipeline at all -- and not for the expected reason (speed): - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns DXGI_ERROR_UNSUPPORTED (0x887A0004). - Drop the flag and WARP creates at FL 11_1, but QueryInterface for ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles. pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so preview and export decode every frame on it. A WARP device would produce none: the retry would only convert a clear startup failure into an obscure ffmpeg one. crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and fails if Windows ever changes them. So there is no fallback, and the answer to "WARP for export or preview only?" is neither. What the compositor does instead is fail in a way the user can act on: - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT) to tell "this adapter has no video decoder" -- the RDP / VM case -- apart from "no FL 11_1 adapter at all", and says which, plus what to do. Export gets this for free: ExportDialog already renders the native message. - That message now reaches the preview too. create_view returns an id long before the render thread can die, so the failure existed only as an eprintln! and the user just saw a black canvas. The thread stores its fatal error in live::Shared, read_frame relays it as an Err on the next pull (~33 ms), and NativeCompositorOverlay renders it in place of the canvas. The WGC capture helper stays hardware-only on purpose: it requests no VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host whose compositor cannot start only produces footage the user can neither edit nor export.
EtienneLescot
added a commit
that referenced
this pull request
Jul 31, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF refactor that extracted `walk_composited_timeline` for the GIF path. The conflict resolution put the WARP branching inside the timeline walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those are MP4-specific. This commit moves the encoder creation back into `run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope) and adds the `if software_frames { ... } else { ... }` branching in the per-frame `on_frame` closure. Also fixes the unclosed `run_gif_bench` delimiter from the conflict resolution and adapts the call site to `export_gif`'s new signature (`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF refactor moved from `(screen, webcam, cursor, out_path, params, dither)` to the slice-2 MP4-shaped signature).
EtienneLescot
added a commit
that referenced
this pull request
Jul 31, 2026
Branch off the CPU-backend seam established by PR #162 and split the compositor crate into per-platform directories. Windows behaviour is byte-identical (81 tests pass) and the macOS stubs that follow this commit in the stack expose the same public surface (Backend, Gpu, Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams, LayerCB...) and Err from every operation. Renames + cfg dispatch: - crates/compositor/src/d3d_windows.rs (from d3d.rs) - crates/compositor/src/cpu_frames_windows.rs (from cpu_frames.rs) - crates/compositor/src/compositor_windows.rs (from compositor.rs) - crates/compositor/src/pipeline_windows.rs (from pipeline.rs) - crates/compositor/src/text_windows.rs (from text.rs) - crates/compositor/wrapper_windows.h (from wrapper.h) lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across the per-platform modules so call-sites stay portable. live.rs: Win32 harness (run_standalone, host_proc, wide, client_size) moved into a #[cfg(windows)] pub mod standalone_harness with a dev-only on Windows). Cargo.toml: macOS deps (metal, objc, block, core-foundation) added under [target.cfg(target_os = macos).dependencies]; windows dep gated to cfg(windows). compositor-view-napi mirrors the gate. crates/.cargo/config.toml: target-specific env block for macOS (LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs). build.rs: selects wrapper_windows.h vs wrapper_macos.h on CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin + -isysroot on macOS, finds MAC_FFMPEG_DIR or vendored thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared. Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot
added a commit
that referenced
this pull request
Jul 31, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into a failing test, and says plainly what the rest is worth. The two backends cannot run on the same machine, so iso cannot be established by comparing two rendered images. What can be — and what is the layer where divergence actually happened on this branch — is the geometry: `plan_frame` is literally the same code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit` pins them for a scene that exercises padding, crop, a displaced PiP camera, corner radii and the zoom, comparing with `to_bits()` rather than an epsilon. It runs in BOTH CI jobs — `rust-macos-compositor-check` and `rust-windows-compositor-check`. If the two platforms ever compute different placements, one of them goes red. That is the guarantee that was being claimed and never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely because nothing compared them. What this test does NOT cover, and must not be read as covering: rasterisation. D3D11 and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical with a max deviation of 3/255 between two backends on the SAME machine, and that is the floor, not the target. Shader parity is held separately, by `shaders.metal` and `shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences came out of that pass, three of them live. The expected values are measured from this code, not chosen. A drift is a divergence to explain, not a tolerance to loosen.
EtienneLescot
added a commit
that referenced
this pull request
Aug 1, 2026
…ing back to WARP PR #162 scoped a WARP retry for the compositor's D3D11 device. Measured, WARP cannot serve this pipeline at all -- and not for the expected reason (speed): - WARP + D3D11_CREATE_DEVICE_VIDEO_SUPPORT does not create. It returns DXGI_ERROR_UNSUPPORTED (0x887A0004). - Drop the flag and WARP creates at FL 11_1, but QueryInterface for ID3D11VideoDevice returns E_NOINTERFACE -- zero decoder profiles. pipeline.rs hands Gpu's device to ffmpeg as the AVD3D11VADeviceContext, so preview and export decode every frame on it. A WARP device would produce none: the retry would only convert a clear startup failure into an obscure ffmpeg one. crates/compositor/tests/warp_device_cannot_decode.rs pins both measurements and fails if Windows ever changes them. So there is no fallback, and the answer to "WARP for export or preview only?" is neither. What the compositor does instead is fail in a way the user can act on: - Gpu::create re-probes on the failure path (same call, minus VIDEO_SUPPORT) to tell "this adapter has no video decoder" -- the RDP / VM case -- apart from "no FL 11_1 adapter at all", and says which, plus what to do. Export gets this for free: ExportDialog already renders the native message. - That message now reaches the preview too. create_view returns an id long before the render thread can die, so the failure existed only as an eprintln! and the user just saw a black canvas. The thread stores its fatal error in live::Shared, read_frame relays it as an Err on the next pull (~33 ms), and NativeCompositorOverlay renders it in place of the canvas. The WGC capture helper stays hardware-only on purpose: it requests no VIDEO_SUPPORT so a WARP device would be creatable there, but recording on a host whose compositor cannot start only produces footage the user can neither edit nor export.
EtienneLescot
added a commit
that referenced
this pull request
Aug 1, 2026
PR #162's CPU backend commit (13e1a31) was applied on top of the GIF refactor that extracted `walk_composited_timeline` for the GIF path. The conflict resolution put the WARP branching inside the timeline walker, which doesn't have `out_w`/`out_h`/`out`/`params` -- those are MP4-specific. This commit moves the encoder creation back into `run_multi_inner` (where `out_w`/`out_h`/`out`/`params` are in scope) and adds the `if software_frames { ... } else { ... }` branching in the per-frame `on_frame` closure. Also fixes the unclosed `run_gif_bench` delimiter from the conflict resolution and adapts the call site to `export_gif`'s new signature (`(clips, out_path, gpu, comp, cfg, params, progress)` -- the GIF refactor moved from `(screen, webcam, cursor, out_path, params, dither)` to the slice-2 MP4-shaped signature).
EtienneLescot
added a commit
that referenced
this pull request
Aug 1, 2026
Branch off the CPU-backend seam established by PR #162 and split the compositor crate into per-platform directories. Windows behaviour is byte-identical (81 tests pass) and the macOS stubs that follow this commit in the stack expose the same public surface (Backend, Gpu, Compositor, Decoder, VideoEncoder, ExportCodec, ClipSource, LiveParams, LayerCB...) and Err from every operation. Renames + cfg dispatch: - crates/compositor/src/d3d_windows.rs (from d3d.rs) - crates/compositor/src/cpu_frames_windows.rs (from cpu_frames.rs) - crates/compositor/src/compositor_windows.rs (from compositor.rs) - crates/compositor/src/pipeline_windows.rs (from pipeline.rs) - crates/compositor/src/text_windows.rs (from text.rs) - crates/compositor/wrapper_windows.h (from wrapper.h) lib.rs cfg-dispatches 'd3d', 'cpu_frames', 'compositor', 'text' across the per-platform modules so call-sites stay portable. live.rs: Win32 harness (run_standalone, host_proc, wide, client_size) moved into a #[cfg(windows)] pub mod standalone_harness with a dev-only on Windows). Cargo.toml: macOS deps (metal, objc, block, core-foundation) added under [target.cfg(target_os = macos).dependencies]; windows dep gated to cfg(windows). compositor-view-napi mirrors the gate. crates/.cargo/config.toml: target-specific env block for macOS (LIBCLANG_PATH not needed, MAC_FFMPEG_DIR falls back via build.rs). build.rs: selects wrapper_windows.h vs wrapper_macos.h on CARGO_CFG_TARGET_OS, sets bindgen --target=aarch64-apple-darwin + -isysroot on macOS, finds MAC_FFMPEG_DIR or vendored thirdparty/ffmpeg-n8.1.2-macos64-lgpl-shared. Stacked on feat/d3d-warp-fallback (PR #162).
EtienneLescot
added a commit
that referenced
this pull request
Aug 1, 2026
"ISO with D3D" was an assertion. This makes the part of it that CAN be measured into a failing test, and says plainly what the rest is worth. The two backends cannot run on the same machine, so iso cannot be established by comparing two rendered images. What can be — and what is the layer where divergence actually happened on this branch — is the geometry: `plan_frame` is literally the same code on both sides, so its 15 outputs must be bit-identical. `plan_frame_is_pinned_bit_for_bit` pins them for a scene that exercises padding, crop, a displaced PiP camera, corner radii and the zoom, comparing with `to_bits()` rather than an epsilon. It runs in BOTH CI jobs — `rust-macos-compositor-check` and `rust-windows-compositor-check`. If the two platforms ever compute different placements, one of them goes red. That is the guarantee that was being claimed and never held: `webcam_shape_code` returned "rounded" on Windows and "rectangle" on macOS for months, and `LiveParams::default()` disagreed on two more fields, precisely because nothing compared them. What this test does NOT cover, and must not be read as covering: rasterisation. D3D11 and Metal will never be bit-identical — PR #162 measured 93-95 % of channels identical with a max deviation of 3/255 between two backends on the SAME machine, and that is the floor, not the target. Shader parity is held separately, by `shaders.metal` and `shaders.hlsl` having been diffed line by line across all 14 modes; nine divergences came out of that pass, three of them live. The expected values are measured from this code, not chosen. A drift is a divergence to explain, not a tolerance to loosen.
EtienneLescot
force-pushed
the
release/v1.8.0
branch
from
August 1, 2026 01:04
4afa3d4 to
2200c7c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Render, decode and encode are three independent axes, and no software rasteriser on any platform also decodes video (WARP here, lavapipe/SwiftShader on Linux, nothing at all on macOS). A fallback needs all three answered, not one driver-type swap:
cpu_frames.rs)h264_amfzero-copyh264_mf/libopenh264via the existingExportCodec::candidates()The seam that keeps output identical: everything the compositor knows about a decoded frame is
compositor::nv12_srvs()/compositor::tex_dims(), which read four fields —data[0](NV12 texture),data[1](array slice),width,height. Fill those andcompositor.rs, every HLSL shader and the scene contract are untouched. A Metal or Vulkan port replacesd3d.rs+cpu_frames.rsand nothing else.Why not the WARP retry this PR scoped
Measured on the reference machine (Ryzen 5 7520U + integrated Radeon, Windows 11):
D3D11_CREATE_DEVICE_VIDEO_SUPPORTDXGI_ERROR_UNSUPPORTED— device does not createQueryInterface(ID3D11VideoDevice)→E_NOINTERFACE, 0 decoder profilespipeline.rshands the device to ffmpeg as theAVD3D11VADeviceContext, so preview and export decode every frame on it. A WARP device would produce zero frames — the retry would have converted a clear startup failure into an obscure ffmpeg one. Locked bycrates/compositor/tests/warp_device_cannot_decode.rs: if that test ever fails, Windows gave WARP video capability and this design should be revisited.The scoped questions, answered: WARP alone is acceptable for neither export nor preview; WGC capture stays hardware-only (see Known gaps); shader cost under WARP is fine except for two multi-tap loops (below).
What shipped
d3d.rs—Backend::{Hardware, Cpu};create(hardware-strict, for tests/goldens),create_auto(production: hardware → CPU fallback), cachedprobe(), anddiagnose()which re-probes minusVIDEO_SUPPORTto separate "this adapter has no video decoder" (RDP / VM / Basic Render Driver — a driver update fixes it) from "no FL 11_1 adapter at all" (structural). That distinction is what the logs and the UI surface.cpu_frames.rs(new) — software decode → swscale → NV12 → upload into one owned dynamic texture → present an AVFrame carrying the 4 seam fields.pipeline.rs—Decodergains the CPU frame source; CPU export skips the encoder pool andVideoEncoder::send_composited()reads composed NV12 straight from the compositor (av_hwframe_transfer_datapresupposes a D3D11 pool WARP cannot create — export died atenc hwdevice initbefore any encoder was tried).live.rs/ napi — the render thread's fatal error is stored and relayed as anErron the nextread_framepull (~33 ms), instead of existing only as aneprintln!under a black canvas. Export usescreate_auto.probeBackend()end to end (addon → service → IPC →useCompositorBackend()hook); the preview carries a small persistent CPU notice and the export dialog warns before the export starts."none"(no addon at all: pure-web dev, jsdom) never warns — that is the normal state in development; only"cpu"is a degraded machine. Tested.errors.previewCompositorUnavailable,cpuCompositor.{notice,exportWarning}.--backend hardware|cpu,--preview,--export; writes a PPM per run so a backend composing black can't post a flattering fps.Measurements (frozen fixture, C1–C8)
h264_amfzero-copy); CPU backend 4.8 fps (h264_mf); CPU forcedlibopenh2644.6 fps. All decode clean underffmpeg -f null -.h264_mfwinning on the CPU backend is a local artefact — Media Foundation picks its MFT independently of our device, so on a machine that has a GPU it still reaches hardware;OPENSCREEN_EXPORT_ENCODER=libopenh264exercises the real last resort.Product decisions
create_auto+probeBackend()+ notices.Known gaps (not fixed here)
is-native-windows-capture-availablechecks only the Windows build and helper-on-disk — never D3D. A host whose helper failscreateD3DDevicegetsavailable: true, commits to native, andstartNativeWindowsRecordingIfAvailablerethrows instead of returningfalse, so thegetDisplayMediafallback is never reached. Not a one-linereturn false: the webcam preview stream has been deliberately released by then, so falling through would silently record screen-only. Fix = re-acquire the stream on the fallback route, or make the probe truthful (helper reports D3D capability before the renderer commits).cfg.bg_blur/cfg.mblur_nare the levers if frame rate ever becomes the complaint.Related issue
n/a — scoping PR that grew into the implementation.
Type of change
Release impact
Desktop impact
Testing
--lib --tests; the 13 Rust doctest failures and thetsconfig.test.jsonerrors are pre-existing on the base branch, verified by stashing).crates/compositor/tests/warp_device_cannot_decode.rspins the WARP probe results above.probeBackend()returns"hardware"on a GPU machine and stays cached; CPU-backend export produces a file that decodes clean end to end.poc-d3d --cfg C1..C8 --backend {hardware,cpu} --frames 300 --repeat 3on the frozen fixture (360 frames, 1920×1080 + 1920×1032, Constrained Baseline 60 fps); each run also writes a PPM frame for pixel inspection.