diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index dc3249c91..923a5c435 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -169,6 +169,40 @@ jobs: env: OPENSCREEN_MAC_HELPER_ARCHS: ${{ matrix.arch }} + # The two steps below are what `npm run build:mac` does and this job did not. + # Windows gets them for free because its job just runs `npm run build:win`, + # which chains fetch:ffmpeg + build:native:compositor; macOS spells its steps + # out (it needs `--dir` plus a hand-rolled DMG and signing) and drifted. The + # result was a .app with no compositor addon — preview and export dead in the + # installed app, silently. `scripts/before-pack.cjs` now refuses to package + # that, so this is also what keeps the job from failing at the pack step. + - name: Cache LGPL ffmpeg tree + uses: actions/cache@v4 + with: + # fetch-ffmpeg-macos.mjs BUILDS ffmpeg from source (~5 min): BtbN ships no + # macOS target and every circulating macOS build is GPL, which would + # relicense this MIT app. The script pins the release and checksums it, so + # keying on the script itself busts the cache when the pin moves. + path: crates/thirdparty + key: ffmpeg-macos-${{ matrix.arch }}-${{ hashFiles('scripts/fetch-ffmpeg-macos.mjs') }} + + - name: Vendor LGPL ffmpeg + run: npm run fetch:ffmpeg:mac + + - name: Cache cargo + compositor build tree + uses: actions/cache@v4 + with: + path: | + ~/.cargo/registry + ~/.cargo/git + crates/target + key: cargo-macos-${{ matrix.arch }}-${{ hashFiles('crates/Cargo.lock') }} + restore-keys: | + cargo-macos-${{ matrix.arch }}- + + - name: Build Metal compositor addon + run: npm run build:native:compositor:mac + - name: Package .app bundle run: npx electron-builder --mac --${{ matrix.arch }} --dir --publish never env: diff --git a/AGENTS.md b/AGENTS.md index b4b873979..1f1a63d40 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,6 +60,7 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi **The HUD widget** (recording controller) - **It is invisible in screenshots by default.** The HUD (and the Notes window) call `setContentProtection(true)` so the recording controls never end up baked into a recording — the same `SetWindowDisplayAffinity` that WGC honours also hides them from *your* screenshots. The window is there, and clicks land, but you are aiming blind at a rectangle you cannot see. Set **`OPENSCREEN_DISABLE_CONTENT_PROTECTION=1`** in the app's environment to turn it off for a session; every skipped window logs a warning. Unset it before recording anything real, or the HUD ends up in the video. +- **On macOS 26+ content protection is auto-disabled, so the HUD *is* visible and screenshottable with no flag.** That OS never displays a content-protected window at all — not just absent from captures, but never painted, leaving a tray icon, a live renderer and nothing on screen (confirmed on macOS 26.5 / Electron 41.2.1). `applyContentProtection` therefore skips the call there and logs a warning per window; the trade-off is that the HUD can appear in recordings on that OS until the ScreenCaptureKit helper excludes our own windows via `SCContentFilter(excludingWindows:)`, which it currently passes as `[]`. `OPENSCREEN_FORCE_CONTENT_PROTECTION=1` re-enables it to re-test against a future Electron. - The HUD is what opens the editor (clapper icon, tooltip *Open Studio*), so without that flag a whole slice of the app is unreachable from automation: killing the app to redeploy a native addon leaves you unable to reopen a project. - Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 600×160). It is **click-through** (`setIgnoreMouseEvents(true, { forward: true })`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. - Control row (left→right): layout preset, **source** button (`Screen`/`Window` → label becomes the picked source), system-audio toggle, mic toggle, **webcam toggle** (shows the detected camera name), cursor-highlight toggle, **record**, notes, open-editor, language, minimize, close. The record button is disabled until a source is chosen (tooltip: "Please select a source to record"). diff --git a/electron/native/whisper-stt/src/main.cpp b/electron/native/whisper-stt/src/main.cpp index 1cc067697..339f759b1 100644 --- a/electron/native/whisper-stt/src/main.cpp +++ b/electron/native/whisper-stt/src/main.cpp @@ -175,14 +175,30 @@ std::string detect_active_backend() { for (size_t i = 0; i < n; ++i) { ggml_backend_dev_t dev = ggml_backend_dev_get(i); if (!dev) continue; + // Only compute devices are candidates. This drops ggml-cpu (type CPU) + // and ggml-blas (type ACCEL — Accelerate on macOS), neither of which is + // the GPU offload this field is meant to report. IGPU is accepted + // because that is how an integrated adapter enumerates. + // `auto`: ggml_backend_dev_type names both a C enum and the accessor + // function, so spelling the type out needs the `enum` tag to disambiguate. + const auto type = ggml_backend_dev_type(dev); + if (type != GGML_BACKEND_DEVICE_TYPE_GPU && type != GGML_BACKEND_DEVICE_TYPE_IGPU) { + continue; + } const char* name = ggml_backend_dev_name(dev); if (!name) continue; std::string s = name; - // ggml-cpu shows up as "CPU" — skip it on the first pass, only return - // it as a last resort. - if (s == "CPU" || s == "cpu") continue; + // These are ggml *device* names, which carry a device index: + // "Vulkan0", "CUDA0", "MTL0". Matching the bare backend title is why + // Metal never matched — ggml-metal's device name is built from + // GGML_METAL_NAME ("MTL"), and the string "Metal" appears nowhere in + // it, so every Apple Silicon run reported `whispercpp-cpu` while the + // Metal backend was in fact bound. `backend` is documented as the + // source of truth for which device ran, so this was the one number a + // consumer could not get right on macOS. if (s.find("Vulkan") != std::string::npos) return "whispercpp-vulkan"; if (s.find("CUDA") != std::string::npos) return "whispercpp-cuda"; + if (s.find("MTL") != std::string::npos) return "whispercpp-metal"; if (s.find("Metal") != std::string::npos) return "whispercpp-metal"; } // Last resort: the CPU device, or a hard "cpu" if ggml has nothing @@ -464,9 +480,26 @@ int main(int argc, char** argv) { } // ---- Emit verbose_json (CT2-compatible shape + backend + timing) ---- + // + // Report the language whisper actually decoded with, not the one the + // request asked for. `language` is the raw request parameter, and the + // renderer sends "auto" on every call (there is no language selector in + // the UI yet), so echoing it back meant `detected_language` never + // carried a language at all: the media stage's "detected language" line + // rendered the literal string "auto", and AxcutTranscript.language was + // set to it via transcribe.ts. whisper_full_lang_id() returns the id + // whisper resolved — the detected one under "auto", and the forced one + // otherwise, which is correct for both paths. + std::string resolved_language = language; + const int lang_id = whisper_full_lang_id(ctx); + if (lang_id >= 0) { + if (const char* lang_str = whisper_lang_str(lang_id)) { + resolved_language = lang_str; + } + } nlohmann::json reply; - reply["language"] = language; - reply["detected_language"] = language; + reply["language"] = resolved_language; + reply["detected_language"] = resolved_language; reply["backend"] = active_backend; reply["timing"] = { {"elapsed_s", elapsed_s}, diff --git a/electron/windows.ts b/electron/windows.ts index 129eb8757..32046fe36 100644 --- a/electron/windows.ts +++ b/electron/windows.ts @@ -19,6 +19,49 @@ const HEADLESS = process.env["HEADLESS"] === "true"; // while it is set WILL contain the HUD. const CONTENT_PROTECTION_DISABLED = process.env["OPENSCREEN_DISABLE_CONTENT_PROTECTION"] === "1"; +// Forces protection back on where it is auto-disabled below, so the macOS +// regression can be re-tested against a future Electron without editing code. +const CONTENT_PROTECTION_FORCED = process.env["OPENSCREEN_FORCE_CONTENT_PROTECTION"] === "1"; + +/** + * macOS 26 (Darwin 25) never displays a window that has had + * `setContentProtection(true)` applied to it. + * + * Not "excludes it from captures" — which is the documented behaviour and the + * whole point — but never paints it for the user either. Confirmed on Darwin + * 25.5 / Electron 41.2.1 with the HUD: tray icon present, renderer alive and + * painting (React mounted, no console errors), `ready-to-show` fired, `show()` + * called, window at valid on-screen coordinates on the main display — and + * nothing on screen. `showMainWindow()` from the tray and the global shortcut + * were equally inert, because the window was already "visible" as far as + * Electron was concerned. Unsetting protection makes it appear instantly; the + * ordering of protect-vs-show makes no difference, so it is the call itself. + * + * `setContentProtection` maps to `NSWindow.sharingType = NSWindowSharingNone`, + * a path Electron has repeatedly churned on (electron/electron#45990, and + * PR #46886 reverting a macOS content-protection refactor). + * + * Gated on the Darwin major version rather than `darwin` wholesale: the + * breakage is only *confirmed* on 25.x, and silently dropping protection on + * older macOS — where it may well work — would be a privacy regression made on + * no evidence. + * + * NOTE: this leaves the HUD capturable on macOS 26. Apple already made that + * partly true regardless — ScreenCaptureKit ignores `sharingType`, so any + * SCK-based recorder (including *ours*, see + * `electron/native/screencapturekit/`) captures these windows anyway. The + * durable fix is to exclude our own windows via `SCContentFilter`'s + * `excludingWindows:`, which that helper currently passes as `[]`. + */ +const CONTENT_PROTECTION_BREAKS_DISPLAY = (() => { + if (process.platform !== "darwin") return false; + // getSystemVersion() reports the *macOS* version on darwin, not the Darwin + // kernel version: measured "26.5.0" here where `os.release()` is "25.5.0". + // So the affected major is 26, and the check must not be fed os.release(). + const macOSMajor = Number.parseInt(process.getSystemVersion().split(".")[0] ?? "", 10); + return Number.isFinite(macOSMajor) && macOSMajor >= 26; +})(); + function applyContentProtection(win: BrowserWindow, label: string) { if (CONTENT_PROTECTION_DISABLED) { console.warn( @@ -28,6 +71,15 @@ function applyContentProtection(win: BrowserWindow, label: string) { ); return; } + if (CONTENT_PROTECTION_BREAKS_DISPLAY && !CONTENT_PROTECTION_FORCED) { + console.warn( + `[content-protection] OFF for the ${label} window — macOS ` + + `${process.getSystemVersion()} never displays a content-protected window, so ` + + "enabling it would make this window permanently invisible. It may therefore appear " + + "in screen captures. Set OPENSCREEN_FORCE_CONTENT_PROTECTION=1 to re-test.", + ); + return; + } win.setContentProtection(true); } diff --git a/package.json b/package.json index 4bb0f3d65..4646e9184 100644 --- a/package.json +++ b/package.json @@ -36,6 +36,7 @@ "build:win:store": "npm run build:native:win && npm run fetch:ffmpeg && npm run build:native:compositor && tsc && vite build && electron-builder --win appx --config.npmRebuild=false", "build:linux": "tsc && vite build && electron-builder --linux AppImage deb pacman --config.npmRebuild=false", "build:whisper-binaries": "bash scripts/build-whisper-stt.sh", + "test:whisper-stt": "node scripts/test-whisper-stt.mjs", "test": "vitest --run", "test:watch": "vitest", "test:cursor-native:win": "node scripts/test-windows-native-cursor.mjs", diff --git a/scripts/before-pack.cjs b/scripts/before-pack.cjs index e64ba780e..c38cea87f 100644 --- a/scripts/before-pack.cjs +++ b/scripts/before-pack.cjs @@ -38,6 +38,99 @@ const SOURCE_PATHS = [ const FIX = "Rebuild it with:\n\n npm run build:native:compositor\n\nor use `npm run build:win`, which does that for you."; +const FIX_MAC = + "Rebuild it with:\n\n npm run build:native:compositor:mac\n\nor use `npm run build:mac`, which does that for you."; + +/** + * Everything that has to be inside `electron/native/bin/darwin-/` for the .app to + * work, keyed by what breaks when it is absent. + * + * This list exists because the macOS deliverable had no guard at all: `beforePack` + * returned early on any non-win32 platform, so a mac package built without the compositor + * addon shipped silently — the preview and the export come up dead, with nothing in any + * log to say why. That is precisely the failure mode the staleness check below was written + * to prevent, and the platform guard was letting it through on the other OS. + * + * `mac.extraResources` ships this directory wholesale (`filter: ["darwin-*​/*"]`), so + * "present here" is the same thing as "present in the installed app". + */ +const MAC_REQUIRED = [ + { + match: (name) => name === "compositor_view.node", + what: "the Metal compositor addon", + breaks: "the preview and every export render nothing", + fix: FIX_MAC, + }, + { + match: (name) => /^libav(codec|format|util)\.\d+\.dylib$/.test(name), + what: "the LGPL ffmpeg dylibs the compositor links", + breaks: "the compositor addon cannot be loaded at all (dyld error at require())", + fix: FIX_MAC, + atLeast: 3, + }, + { + match: (name) => name === "whisper-stt-server", + what: "the whisper.cpp STT helper", + breaks: "transcription and captions fail with a developer error shown to end users", + fix: "Build it with:\n\n npm run build:whisper-binaries\n\nor stage CI's with `bash scripts/stage-whisper-stt.sh darwin-`.", + }, + { + match: (name) => /^libggml.*\.dylib$/.test(name), + what: "the ggml backend dylibs the STT helper links", + breaks: "whisper-stt-server dies in dyld before main(), so STT times out with no diagnostic", + fix: "Build it with:\n\n npm run build:whisper-binaries", + atLeast: 1, + }, + { + match: (name) => name === "openscreen-screencapturekit-helper", + what: "the ScreenCaptureKit capture helper", + breaks: "native screen capture is unavailable", + fix: "Build it with:\n\n npm run build:native:mac", + }, +]; + +/** electron-builder passes `context.arch` as a numeric enum; map it to our directory tag. */ +function archTagFor(context) { + const BY_INDEX = { 0: "ia32", 1: "x64", 2: "armv7l", 3: "arm64", 4: "universal" }; + const name = BY_INDEX[context?.arch]; + return name && name !== "universal" ? name : process.arch; +} + +function checkMacNativePayload(context) { + const tag = `darwin-${archTagFor(context)}`; + const dir = path.join(ROOT, "electron", "native", "bin", tag); + if (!fs.existsSync(dir)) { + throw new Error( + `Refusing to package: ${path.relative(ROOT, dir)} does not exist, so the .app would ` + + "ship with no native modules at all.\n\n" + + `${FIX_MAC}\n\nThe STT helper and the capture helper are separate builds — see\n` + + "technical-documentation/engineering/build-and-packaging.md.", + ); + } + + const present = fs.readdirSync(dir); + const missing = MAC_REQUIRED.filter( + (req) => present.filter((name) => req.match(name)).length < (req.atLeast ?? 1), + ); + if (missing.length === 0) { + return; + } + + const detail = missing + .map( + (req) => + ` - ${req.what}\n without it: ${req.breaks}\n ${req.fix.replace(/\n+/g, " ")}`, + ) + .join("\n"); + throw new Error( + `Refusing to package an incomplete macOS payload.\n\n` + + ` looked in: ${path.relative(ROOT, dir)}\n\n` + + `Missing:\n${detail}\n\n` + + "Every one of these fails silently or as an unactionable timeout in the installed\n" + + "app, which is why this is a hard error at pack time rather than a warning.", + ); +} + /** Newest mtime under `target` (file or directory), or 0 if it does not exist. */ function newestMtimeMs(target) { let stat; @@ -56,14 +149,14 @@ function newestMtimeMs(target) { return newest; } -function checkCompositorAddonFreshness() { - if (!fs.existsSync(ADDON)) { +function checkCompositorAddonFreshness(addon = ADDON, fix = FIX, label = "D3D11") { + if (!fs.existsSync(addon)) { throw new Error( - `Refusing to package: the D3D11 compositor addon is missing.\n\n expected: ${ADDON}\n\n${FIX}`, + `Refusing to package: the ${label} compositor addon is missing.\n\n expected: ${addon}\n\n${fix}`, ); } - const addonMs = fs.statSync(ADDON).mtimeMs; + const addonMs = fs.statSync(addon).mtimeMs; const stale = SOURCE_PATHS.map((source) => ({ source, ms: newestMtimeMs(source) })).filter( (entry) => entry.ms > addonMs, ); @@ -73,29 +166,51 @@ function checkCompositorAddonFreshness() { const newest = stale.reduce((a, b) => (a.ms > b.ms ? a : b)); throw new Error( - "Refusing to package a stale D3D11 compositor addon.\n\n" + + `Refusing to package a stale ${label} compositor addon.\n\n` + + ` addon: ${path.relative(ROOT, addon)}\n` + ` addon built: ${new Date(addonMs).toISOString()}\n` + ` newer source: ${path.relative(ROOT, newest.source)} (${new Date(newest.ms).toISOString()})\n\n` + "Packaging this would silently ship an addon that ignores newer scene fields\n" + "(they are #[serde(default)], so it falls back instead of erroring).\n\n" + - FIX, + fix, ); } exports.default = async function beforePack(context) { - // The compositor addon is Windows-only; skip when packing any other target. const platform = context?.electronPlatformName ?? process.platform; - if (platform !== "win32") { + if (platform === "win32") { + checkCompositorAddonFreshness(); return; } - checkCompositorAddonFreshness(); + if (platform === "darwin") { + // The addon that actually ships is the arch-tagged copy under + // electron/native/bin/ (mac.extraResources), not the dev copy this hook used to + // be the sole guardian of — so that is the one whose freshness matters. + const tag = `darwin-${archTagFor(context)}`; + const shipped = path.join(ROOT, "electron", "native", "bin", tag, "compositor_view.node"); + checkMacNativePayload(context); + checkCompositorAddonFreshness(shipped, FIX_MAC, "Metal"); + return; + } + // Linux ships no native addon of its own; nothing to assert. }; // Runnable on its own for debugging: `node scripts/before-pack.cjs` if (require.main === module) { try { - checkCompositorAddonFreshness(); - console.log("compositor addon is up to date with its Rust sources."); + if (process.platform === "darwin") { + checkMacNativePayload({ arch: undefined }); + const tag = `darwin-${process.arch}`; + checkCompositorAddonFreshness( + path.join(ROOT, "electron", "native", "bin", tag, "compositor_view.node"), + FIX_MAC, + "Metal", + ); + console.log(`macOS native payload complete in electron/native/bin/${tag}, addon up to date.`); + } else { + checkCompositorAddonFreshness(); + console.log("compositor addon is up to date with its Rust sources."); + } } catch (err) { console.error(err.message); process.exit(1); diff --git a/scripts/build-macos-compositor-addon.mjs b/scripts/build-macos-compositor-addon.mjs index a66bb2803..200b9cd87 100644 --- a/scripts/build-macos-compositor-addon.mjs +++ b/scripts/build-macos-compositor-addon.mjs @@ -190,14 +190,47 @@ function vendorFfmpegDylibs(nodePath, ffmpegDir) { execFileSync("codesign", ["--force", "--sign", "-", target]); } - const remaining = execFileSync("otool", ["-L", nodePath], { encoding: "utf8" }) - .split("\n") - .map((l) => l.trim().split(" ")[0]) - .filter((p) => p.startsWith("/") && /lib(av|sw)/.test(p)); - if (remaining.length > 0) { - throw new Error(`Still absolute after rewriting: ${remaining.join(", ")}`); + // cargo stamps the cdylib's own id with the absolute path it was built at + // (`crates/target/release/deps/libcompositor_view.dylib`). Nothing resolves + // through it — `require()` dlopens the file by path and never reads + // LC_ID_DYLIB — so this is not a load failure. It is a build-machine path, + // complete with the builder's home directory and checkout name, shipped + // inside a release artefact: it defeats reproducible builds and leaks a + // filesystem layout to anyone who runs `otool -D` on the installed app. + execFileSync("install_name_tool", ["-id", `@rpath/${path.basename(nodePath)}`, nodePath]); + execFileSync("codesign", ["--force", "--sign", "-", nodePath]); + + // The old check only grepped for `lib(av|sw)`, so it could not have caught + // the id above — nor any future non-ffmpeg dependency that arrives absolute. + // Assert the real invariant instead: nothing outside the OS's own prefixes + // may be referenced by absolute path. + const absolutePaths = (file) => { + const id = execFileSync("otool", ["-D", file], { encoding: "utf8" }) + .split("\n") + .slice(1) // first line is the filename echoed back + .map((l) => l.trim()) + .filter(Boolean); + const deps = execFileSync("otool", ["-L", file], { encoding: "utf8" }) + .split("\n") + .slice(1) // ditto + .map((l) => l.trim().split(" ")[0]) + .filter(Boolean); + return [...id, ...deps].filter( + (p) => p.startsWith("/") && !p.startsWith("/usr/lib/") && !p.startsWith("/System/"), + ); + }; + + for (const file of [nodePath, ...names.map((n) => path.join(outDir, n))]) { + const remaining = absolutePaths(file); + if (remaining.length > 0) { + throw new Error( + `${path.basename(file)} still references build-machine paths after rewriting: ` + + remaining.join(", "), + ); + } } console.log(`Vendored ${names.length} ffmpeg dylibs next to ${path.basename(nodePath)}`); + console.log(`No absolute build-machine paths remain in ${path.basename(nodePath)} or its dylibs`); } /** diff --git a/scripts/build-whisper-stt.sh b/scripts/build-whisper-stt.sh index 2e8ec5ffe..16d6a4f7e 100644 --- a/scripts/build-whisper-stt.sh +++ b/scripts/build-whisper-stt.sh @@ -87,6 +87,59 @@ backend_flag_for_host() { esac } +# --------------------------------------------------------------------------- +# macOS: make the staged tree relocatable. +# +# CMake links the helper and every ggml/whisper dylib against `@rpath/...` and +# then bakes a single absolute LC_RPATH pointing at its own build tree +# (`.cache/whisper-stt-build/build--/bin`). That resolves on the +# machine that ran the build and nowhere else: delete the build cache, or +# download the CI artifact onto a different runner, and dyld cannot find +# libwhisper at all — the process aborts (SIGABRT, exit 134) before main(). +# +# This is the macOS twin of the Windows 0xC0000135 incident that shipped in +# 1.8.0-rc.1..rc.3 (see the OpenSSL note in the CMakeLists). Swapping the +# absolute rpath for `@loader_path` makes each binary look for its siblings +# next to itself, which is exactly how the staged directory and the packaged +# `resources/electron/native/bin//` are laid out. +# +# Editing a Mach-O header invalidates its code signature, and arm64 refuses to +# execute an unsigned-but-modified image ("killed: 9"), so each patched file is +# re-signed ad-hoc afterwards. +# --------------------------------------------------------------------------- +relocate_macos_rpaths() { + local out_dir="$1" + + local f rp + for f in "${out_dir}"/*; do + # Symlinks share the payload of their target; patching the target is + # enough, and codesign would follow the link and sign it twice. + [[ -L "${f}" ]] && continue + [[ -f "${f}" ]] || continue + # Mach-O only: skips *.metal shader sidecars and anything else non-binary. + file -b "${f}" | grep -q "Mach-O" || continue + + # Drop every *absolute* LC_RPATH. An absolute rpath in a shipped artifact + # is always a build-machine path; relative ones (@loader_path, + # @executable_path) are already correct and must survive. + while read -r rp; do + [[ "${rp}" == /* ]] || continue + install_name_tool -delete_rpath "${rp}" "${f}" 2>/dev/null || true + done < <(otool -l "${f}" | awk '/cmd LC_RPATH/{n=1;next} n&&/^ *path /{print $2;n=0}') + + # Idempotent: a second run would otherwise stack duplicate rpaths. + if ! otool -l "${f}" | awk '/cmd LC_RPATH/{n=1;next} n&&/^ *path /{print $2;n=0}' | + grep -qx "@loader_path"; then + install_name_tool -add_rpath "@loader_path" "${f}" + fi + + codesign --force --sign - --timestamp=none "${f}" 2>/dev/null || + echo "[whisper-stt] WARN: could not re-sign ${f##*/}" >&2 + done + + echo "[whisper-stt] rewrote macOS rpaths to @loader_path in ${out_dir}" +} + build_variant() { local variant_name="$1" shift @@ -109,7 +162,10 @@ build_variant() { ${extra_cmake_flags[@]+"${extra_cmake_flags[@]}"} echo "[whisper-stt] building ${variant_name}" - cmake --build "${build_dir}" --config Release -j "$(nproc 2>/dev/null || echo 4)" + # ponytail: macOS has no `nproc` (it is GNU coreutils), so this silently built + # with -j4 on an 8-core Mac. `sysctl -n hw.ncpu` is the BSD equivalent. + cmake --build "${build_dir}" --config Release \ + -j "$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)" local bin_name="whisper-stt-server" if [[ "${OS_ARCH}" == win32-* ]]; then @@ -152,14 +208,36 @@ build_variant() { # Stage any shared libraries / backend sidecars that CMake produced. # Copy everything that looks like a ggml/whisper shared library, plus any # .metal shader files (only produced when Metal is not embedded). + # + # ponytail: the pattern list used to be `ggml*.*|whisper.*|libwhisper.*`, + # which only ever matched on Windows. CMake names shared libraries + # `ggml-base.dll` on Windows but `libggml-base.dylib` / `libggml-base.so.0` + # everywhere else, so on macOS and Linux every ggml sidecar was skipped and + # only libwhisper was staged — leaving a helper that cannot resolve + # @rpath/libggml.0.dylib and dies in dyld before main(). The `lib` prefix and + # the `.so.` version suffixes are what the globs below add. + # + # -a preserves the symlink farm (libggml.dylib -> libggml.0.dylib -> + # libggml.0.15.1.dylib); plain `cp` dereferences each one into a full copy of + # the same payload, which tripled the staged size for no benefit. local found_libs=0 for lib_dir in "${search_dirs[@]}"; do if [[ -d "${lib_dir}" ]]; then for f in "${lib_dir}"/*; do - if [[ -f "${f}" ]]; then + # -f follows symlinks (true for both link and target); -L catches the + # links themselves so the farm is staged intact. + if [[ -f "${f}" || -L "${f}" ]]; then + # Patterns, in order: Windows DLLs; macOS dylibs; Linux .so plus its + # versioned forms (libggml-base.so.0, libggml-base.so.0.15.1); Metal + # shader sidecars. A comment cannot be placed inside a `case` pattern + # list — a `| \` continuation would swallow it as part of the pattern. case "${f##*/}" in - ggml*.*|whisper.*|whisper.dylib|whisper.dll|libwhisper.*|*.metal) - cp "${f}" "${OUT_DIR}/" + ggml*.dll|whisper.dll|parakeet.dll|\ +libggml*.dylib|libwhisper*.dylib|libparakeet*.dylib|\ +libggml*.so|libggml*.so.*|libwhisper*.so|libwhisper*.so.*|\ +libparakeet*.so|libparakeet*.so.*|\ +*.metal) + cp -a "${f}" "${OUT_DIR}/" found_libs=1 ;; esac @@ -168,6 +246,21 @@ build_variant() { fi done + # Fail here rather than in the installer. `found_libs` was computed and then + # never read, so a staging glob that matched nothing — which is precisely what + # happened on macOS and Linux — still exited 0 and produced a green build. + if [[ "${found_libs}" -eq 0 ]]; then + echo "FATAL: staged no shared libraries alongside ${out_bin_name}." >&2 + echo " Searched: ${search_dirs[*]}" >&2 + echo " The helper links ggml/whisper as shared libraries; without them" >&2 + echo " it cannot start. Check the sidecar glob in this script." >&2 + exit 1 + fi + + if [[ "${OS_ARCH}" == darwin-* ]]; then + relocate_macos_rpaths "${OUT_DIR}" + fi + echo "[whisper-stt] built ${variant_name} -> ${OUT_DIR}/${out_bin_name}" ls -la "${OUT_DIR}" } diff --git a/scripts/stage-whisper-stt.sh b/scripts/stage-whisper-stt.sh index ffe9b45a7..37883d34e 100755 --- a/scripts/stage-whisper-stt.sh +++ b/scripts/stage-whisper-stt.sh @@ -86,9 +86,19 @@ if [ "${TAG#win32}" != "${TAG}" ]; then else MINIMAL_PATH="/usr/bin:/bin" fi +# `timeout` is GNU coreutils: absent from a stock macOS runner. Unqualified, it +# made the whole check evaporate there — `env: timeout: No such file or +# directory` is itself output, so the "did it print anything" test below passed +# without ever starting the binary. Use it only when it exists. +if command -v timeout >/dev/null 2>&1; then + TIMEOUT_CMD="timeout 60" +else + TIMEOUT_CMD="" +fi # A bad --model makes it fail fast; we only care that it got far enough to speak. set +e -LOAD_OUT="$(cd "${DEST}" && timeout 60 env PATH="${MINIMAL_PATH}" \ +# shellcheck disable=SC2086 # TIMEOUT_CMD is deliberately word-split (or empty). +LOAD_OUT="$(cd "${DEST}" && ${TIMEOUT_CMD} env PATH="${MINIMAL_PATH}" \ "./$(basename "${BIN}")" --model "__staging_load_check__" 2>&1)" LOAD_CODE=$? set -e @@ -98,6 +108,26 @@ if [ -z "${LOAD_OUT}" ]; then echo " Windows exit 3221225781 = 0xC0000135 STATUS_DLL_NOT_FOUND." >&2 exit 1 fi + +# "It printed something" is a Windows-shaped test: the loader there dies mute. +# dyld and ld.so are chatty, so on macOS/Linux a helper that never reached +# main() still satisfies the check above — which is exactly how a macOS build +# missing every libggml*.dylib passed staging. Demand the marker that main() +# itself prints (electron/native/whisper-stt/src/main.cpp), so the gate proves +# execution rather than mere noise. +case "${LOAD_OUT}" in + *"[whisper-stt] boot:"*) ;; + *) + echo "FATAL: $(basename "${BIN}") never reached main() (exit ${LOAD_CODE})." >&2 + echo " Expected its '[whisper-stt] boot:' line; got:" >&2 + printf '%s\n' "${LOAD_OUT}" | sed 's/^/ /' | head -n 12 >&2 + echo " A dynamic-loader error here means a sidecar is missing from the" >&2 + echo " artifact, or the binary carries an absolute rpath/RUNPATH into" >&2 + echo " the build machine's tree. Both are bugs in" >&2 + echo " scripts/build-whisper-stt.sh's staging step." >&2 + exit 1 + ;; +esac echo "Load check OK (exit ${LOAD_CODE}): $(printf '%s' "${LOAD_OUT}" | head -n 1)" echo "Staged $(basename "${BIN}") + $(( $(ls -1 "${DEST}" | wc -l) - 1 )) sidecar(s) -> ${DEST}" diff --git a/scripts/test-whisper-stt.mjs b/scripts/test-whisper-stt.mjs new file mode 100644 index 000000000..7e52bac2b --- /dev/null +++ b/scripts/test-whisper-stt.mjs @@ -0,0 +1,363 @@ +// Round-trips a real WAV through a real `whisper-stt-server` and checks the +// response against the invariants in +// technical-documentation/architecture/transcription-and-captions.md. +// +// WHY THIS EXISTS. The helper's unit tests mock `fetch`, so every assertion +// about the wire format was made against a hand-written fixture rather than the +// binary. Three defects lived in that blind spot at once on macOS: no ggml +// dylib was staged next to the executable, the executable carried an absolute +// rpath into the build tree, and `backend`/`detected_language` reported values +// the helper had never actually resolved. All three are things only a real +// request can see. +// +// Usage: +// node scripts/test-whisper-stt.mjs # macOS: synthesizes speech with `say` +// node scripts/test-whisper-stt.mjs --wav a.wav # any platform: bring your own clip +// node scripts/test-whisper-stt.mjs --wav a.wav --ref "expected words" +// node scripts/test-whisper-stt.mjs --language fr # force a language instead of auto +// +// Env overrides: +// OPENSCREEN_WHISPER_SERVER_EXE helper binary (default: the staged one for this host) +// OPENSCREEN_WHISPER_MODEL GGML model (default: the userData cache location) + +import { spawn, spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const ROOT = path.join(__dirname, ".."); + +const argOf = (flag) => { + const i = process.argv.indexOf(flag); + return i >= 0 && i + 1 < process.argv.length ? process.argv[i + 1] : null; +}; + +const TAG = `${process.platform}-${process.arch}`; +const BIN = + process.env.OPENSCREEN_WHISPER_SERVER_EXE ?? + path.join( + ROOT, + "electron", + "native", + "bin", + TAG, + process.platform === "win32" ? "whisper-stt-server.exe" : "whisper-stt-server", + ); + +const MODEL = process.env.OPENSCREEN_WHISPER_MODEL ?? defaultModelPath(); +const LANGUAGE = argOf("--language") ?? "auto"; +const WAV_ARG = argOf("--wav"); +const REF_ARG = argOf("--ref"); + +/** + * Mirrors `SttManager`'s cache location: `app.getPath("userData")/stt-models` + * ([electron/stt/index.ts:69](../electron/stt/index.ts)). + * + * The leaf of that userData path is `app.getName()`, which is package.json's + * `name` ("openscreen") in dev and the productName once packaged — *not* + * "Electron", which is only what a bare `electron .` with no app name would + * use. Probe the plausible names and take the one that exists, so this works + * whether the model was cached by the app or by a bare helper run. + */ +function defaultModelPath() { + const file = path.join("stt-models", "whisper-ggml", "ggml-small-q8_0.bin"); + const roots = + process.platform === "darwin" + ? [path.join(os.homedir(), "Library", "Application Support")] + : process.platform === "win32" + ? [process.env.APPDATA ?? ""] + : [process.env.XDG_CONFIG_HOME || path.join(os.homedir(), ".config")]; + const candidates = roots.flatMap((root) => + ["openscreen", "Openscreen", "Electron"].map((appName) => path.join(root, appName, file)), + ); + return candidates.find((p) => fs.existsSync(p)) ?? candidates[0]; +} + +const failures = []; +const check = (ok, label, detail = "") => { + console.log(` ${ok ? "PASS" : "FAIL"} ${label}${detail ? ` — ${detail}` : ""}`); + if (!ok) failures.push(label); +}; + +/** Minimal RIFF/WAVE parse: enough to assert the clip is what the helper expects. */ +function readWavMeta(file) { + const buf = fs.readFileSync(file); + if (buf.toString("ascii", 0, 4) !== "RIFF" || buf.toString("ascii", 8, 12) !== "WAVE") { + throw new Error(`${file} is not a RIFF/WAVE file`); + } + let offset = 12; + let fmt = null; + let dataBytes = null; + while (offset + 8 <= buf.length) { + const id = buf.toString("ascii", offset, offset + 4); + const size = buf.readUInt32LE(offset + 4); + if (id === "fmt ") { + fmt = { + channels: buf.readUInt16LE(offset + 10), + sampleRate: buf.readUInt32LE(offset + 12), + bitsPerSample: buf.readUInt16LE(offset + 22), + }; + } else if (id === "data") { + dataBytes = size; + } + offset += 8 + size + (size % 2); + } + if (!fmt || dataBytes == null) throw new Error(`${file} has no fmt/data chunk`); + const bytesPerFrame = (fmt.bitsPerSample / 8) * fmt.channels; + return { ...fmt, durationSec: dataBytes / bytesPerFrame / fmt.sampleRate }; +} + +/** macOS-only: synthesize a clip with a known transcript so WER is meaningful. */ +function synthesizeWav() { + const text = + "And so my fellow Americans, ask not what your country can do for you, " + + "ask what you can do for your country."; + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "whisper-stt-test-")); + const aiff = path.join(dir, "speech.aiff"); + const wav = path.join(dir, "speech.wav"); + for (const [cmd, args] of [ + ["say", ["-o", aiff, text]], + ["afconvert", ["-f", "WAVE", "-d", "LEI16@16000", "-c", "1", aiff, wav]], + ]) { + const r = spawnSync(cmd, args, { stdio: "inherit" }); + if (r.status !== 0) throw new Error(`${cmd} failed (status ${r.status})`); + } + return { wav, ref: text, cleanup: () => fs.rmSync(dir, { recursive: true, force: true }) }; +} + +const normalize = (s) => + s + .toLowerCase() + .replace(/[^\p{L}\p{N}\s']/gu, " ") + .split(/\s+/) + .filter(Boolean); + +/** Word error rate: Levenshtein over word arrays, (S+D+I)/N. */ +function wer(refText, hypText) { + const ref = normalize(refText); + const hyp = normalize(hypText); + if (ref.length === 0) return hyp.length === 0 ? 0 : 1; + let prev = Array.from({ length: hyp.length + 1 }, (_, j) => j); + for (let i = 1; i <= ref.length; i++) { + const cur = [i]; + for (let j = 1; j <= hyp.length; j++) { + cur[j] = Math.min( + prev[j] + 1, + cur[j - 1] + 1, + prev[j - 1] + (ref[i - 1] === hyp[j - 1] ? 0 : 1), + ); + } + prev = cur; + } + return prev[hyp.length] / ref.length; +} + +async function waitForReady(baseUrl, timeoutMs = 120_000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + try { + const res = await fetch(baseUrl, { method: "GET" }); + if (res.ok) return; + } catch { + // not up yet + } + await new Promise((r) => setTimeout(r, 250)); + } + throw new Error(`helper did not become ready within ${timeoutMs}ms`); +} + +async function main() { + for (const [label, p] of [ + ["helper binary", BIN], + ["GGML model", MODEL], + ]) { + if (!fs.existsSync(p)) { + console.error(`FATAL: ${label} not found at ${p}`); + if (label === "helper binary") { + console.error(" Build it: bash scripts/build-whisper-stt.sh"); + } else { + console.error( + " Run a transcription in the app once, or set OPENSCREEN_WHISPER_MODEL.", + ); + } + process.exit(1); + } + } + + let wavPath = WAV_ARG; + let refText = REF_ARG; + // No-op unless we synthesize a clip into a temp dir that needs removing. + let cleanup = () => undefined; + if (!wavPath) { + if (process.platform !== "darwin") { + console.error("FATAL: --wav is required off macOS (no `say` to synthesize with)."); + process.exit(2); + } + const made = synthesizeWav(); + wavPath = made.wav; + refText = refText ?? made.ref; + cleanup = made.cleanup; + } + + const meta = readWavMeta(wavPath); + console.log(`\nclip : ${wavPath}`); + console.log( + `format : ${meta.channels}ch ${meta.sampleRate}Hz ${meta.bitsPerSample}-bit, ` + + `${meta.durationSec.toFixed(2)}s`, + ); + console.log(`helper : ${BIN}`); + console.log(`model : ${MODEL}\n`); + + const port = 20100 + Math.floor(process.pid % 400); + const child = spawn( + BIN, + [ + "--model", + MODEL, + "--port", + String(port), + "--host", + "127.0.0.1", + "--threads", + String(Math.max(1, os.cpus().length)), + ], + { stdio: ["ignore", "pipe", "pipe"] }, + ); + let stderr = ""; + child.stderr.on("data", (d) => { + stderr += String(d); + }); + child.on("exit", (code, signal) => { + if (code !== 0 && code !== null) { + console.error(`\nhelper exited early (code ${code}, signal ${signal}):`); + console.error(stderr.split("\n").slice(-15).join("\n")); + } + }); + + let json; + try { + await waitForReady(`http://127.0.0.1:${port}/`); + const form = new FormData(); + form.append("file", new Blob([fs.readFileSync(wavPath)]), path.basename(wavPath)); + form.append("language", LANGUAGE); + form.append("response_format", "verbose_json"); + const res = await fetch(`http://127.0.0.1:${port}/inference`, { + method: "POST", + body: form, + }); + if (!res.ok) throw new Error(`/inference returned ${res.status}: ${await res.text()}`); + json = await res.json(); + } finally { + child.kill(); + cleanup(); + } + + const segments = json.segments ?? []; + const words = segments.flatMap((s) => s.words ?? []); + const text = segments + .map((s) => s.text ?? "") + .join("") + .trim(); + + console.log(`text : ${text}\n`); + console.log("assertions:"); + + check(segments.length > 0, "response carries phrase segments", `${segments.length}`); + check(words.length > 0, "response carries per-word timings", `${words.length} words`); + + // The DTW guardrail lives in the helper; its stderr line is the only place + // the result is reported, and a FAIL there is a 500 rather than a bad body. + check(/§4\.1 guardrail: PASS/.test(stderr), "DTW guardrail passed in the helper"); + + // `backend` is documented as the source of truth for the device that ran. + // Metal reported as CPU is the exact bug this catches: ggml's Metal *device* + // is named "MTL0", so a matcher looking for "Metal" never fires. + const gpuExpected = + (process.platform === "darwin" && process.arch === "arm64") || process.platform === "win32"; + check( + typeof json.backend === "string" && json.backend.startsWith("whispercpp-"), + "backend is a contract value", + String(json.backend), + ); + if (gpuExpected) { + check( + json.backend !== "whispercpp-cpu", + "backend reports GPU offload on a GPU-capable host", + String(json.backend), + ); + } + + // "auto" is a request value, never a response value: the helper must report + // the language whisper resolved. + check( + typeof json.detected_language === "string" && + json.detected_language !== "auto" && + json.detected_language.length > 0, + "detected_language is a resolved language, not the request echo", + String(json.detected_language), + ); + + // Contract invariant: absolute seconds in the source recording, [0, duration). + // + // These are checked separately rather than as one `every()`. Bundled, a + // failure could not say which property broke: a real French clip tripped the + // combined check and read as "word times outside the clip" when nothing was + // outside the clip at all — three punctuation tokens had end < start. + const tolerance = 0.5; // whisper rounds segment ends up to the chunk edge + check( + words.every((w) => w.start >= 0), + "word starts are non-negative", + ); + check( + words.every((w) => w.end <= meta.durationSec + tolerance), + "word ends lie within the clip", + `clip is ${meta.durationSec.toFixed(2)}s`, + ); + check( + words.every((w, i) => i === 0 || w.start >= words[i - 1].start), + "word starts are monotonic non-decreasing", + ); + + // NOT a failure. whisper.cpp gives the last word of a segment the segment's + // own t1 as its end, while the word's DTW start runs 80–150 ms late — so a + // token emitted near the boundary (typically standalone punctuation) can land + // after it and invert. The contract's [startSec, endSec) guarantee is enforced + // one layer up, deliberately: whisperServer.ts clamps to + // `Math.max(startSec + 0.02, endSec)` and snapWordBoundaries.ts keeps + // "degenerate words (whisper sometimes reports end <= start) non-empty". This + // harness speaks to the raw helper, below that clamp, so it reports the count + // as information instead of asserting on it. + const inverted = words.filter((w) => w.end < w.start); + if (inverted.length > 0) { + console.log( + ` info ${inverted.length}/${words.length} raw word(s) have end < start ` + + `(${inverted.map((w) => JSON.stringify(w.word)).join(", ")}) — expected at segment ` + + "boundaries; whisperServer.ts clamps these before they reach the document.", + ); + } + + if (refText) { + const rate = wer(refText, text); + check(rate <= 0.15, "WER within tolerance", `${rate.toFixed(4)}`); + } + + if (json.timing) { + console.log( + `\ntiming : ${json.timing.elapsed_s?.toFixed?.(2)}s for ` + + `${json.timing.audio_s?.toFixed?.(2)}s audio (rtf ${json.timing.rtf?.toFixed?.(3)})`, + ); + } + + if (failures.length > 0) { + console.error(`\n${failures.length} check(s) failed: ${failures.join("; ")}`); + process.exit(1); + } + console.log("\nAll checks passed."); +} + +main().catch((err) => { + console.error(`\nFATAL: ${err.message}`); + process.exit(1); +}); diff --git a/src/i18n/locales/ar/settings.json b/src/i18n/locales/ar/settings.json index 242fd8c82..39e893a6c 100644 --- a/src/i18n/locales/ar/settings.json +++ b/src/i18n/locales/ar/settings.json @@ -245,7 +245,7 @@ "help": "نص مجمّع لكل المقاطع على المخطط الزمني. اضغط Backspace / Delete على كلمة أو تحديد لتمييزه كمتخطّى (بالأحمر). مرّر المؤشر فوق المقطع الأحمر لاستعادته.", "noClips": "لا توجد مقاطع بعد", "noTranscript": "لا يوجد نص بعد", - "whisperHint": "يستخدم التفريغ النصي Whisper محليًا — يعمل داخل متصفحك، ولا تغادر أي بيانات جهازك.", + "whisperHint": "يستخدم التفريغ النصي Whisper محليًا — يعمل على حاسوبك، ولا تغادر أي بيانات جهازك.", "transcribeNow": "فرّغ النص الآن", "transcribing": "جارٍ التفريغ…", "clipLabel": "المقطع {{index}}", diff --git a/src/i18n/locales/en/settings.json b/src/i18n/locales/en/settings.json index 19acd784f..8ebfb79ed 100644 --- a/src/i18n/locales/en/settings.json +++ b/src/i18n/locales/en/settings.json @@ -245,7 +245,7 @@ "help": "Aggregated transcript of every clip on the timeline. Backspace / Delete a word or selection to mark it as skipped (red). Hover a red span to restore it.", "noClips": "No clips yet", "noTranscript": "No transcript yet", - "whisperHint": "Transcribe uses local Whisper — runs in your browser, no data leaves the device.", + "whisperHint": "Transcribe uses local Whisper — runs on your computer, no data leaves the device.", "transcribeNow": "Transcribe now", "transcribing": "Transcribing…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/es/settings.json b/src/i18n/locales/es/settings.json index c0a5a113c..68bd83ec0 100644 --- a/src/i18n/locales/es/settings.json +++ b/src/i18n/locales/es/settings.json @@ -245,7 +245,7 @@ "help": "Transcripción agregada de todos los clips de la línea de tiempo. Retroceso / Supr sobre una palabra o selección la marca como omitida (en rojo). Pasa el cursor sobre un fragmento rojo para restaurarlo.", "noClips": "Aún no hay clips", "noTranscript": "Aún no hay transcripción", - "whisperHint": "La transcripción usa Whisper local: se ejecuta en tu navegador y ningún dato sale del dispositivo.", + "whisperHint": "La transcripción usa Whisper local: se ejecuta en tu equipo y ningún dato sale del dispositivo.", "transcribeNow": "Transcribir ahora", "transcribing": "Transcribiendo…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/fr/settings.json b/src/i18n/locales/fr/settings.json index e78e96a83..13f650f5f 100644 --- a/src/i18n/locales/fr/settings.json +++ b/src/i18n/locales/fr/settings.json @@ -245,7 +245,7 @@ "help": "Transcription agrégée de tous les clips de la timeline. Retour arrière / Suppr sur un mot ou une sélection le marque comme ignoré (en rouge). Survolez un passage rouge pour le restaurer.", "noClips": "Aucun clip pour l'instant", "noTranscript": "Aucune transcription pour l'instant", - "whisperHint": "La transcription utilise Whisper en local — tout s'exécute dans votre navigateur, aucune donnée ne quitte l'appareil.", + "whisperHint": "La transcription utilise Whisper en local — tout s'exécute sur votre ordinateur, aucune donnée ne quitte l'appareil.", "transcribeNow": "Transcrire maintenant", "transcribing": "Transcription…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/it/settings.json b/src/i18n/locales/it/settings.json index 6361208a3..9f6f82393 100644 --- a/src/i18n/locales/it/settings.json +++ b/src/i18n/locales/it/settings.json @@ -245,7 +245,7 @@ "help": "Trascrizione aggregata di tutti i clip sulla timeline. Backspace / Canc su una parola o selezione la segna come saltata (in rosso). Passa sopra un tratto rosso per ripristinarlo.", "noClips": "Ancora nessun clip", "noTranscript": "Ancora nessuna trascrizione", - "whisperHint": "La trascrizione usa Whisper in locale: gira nel tuo browser, nessun dato lascia il dispositivo.", + "whisperHint": "La trascrizione usa Whisper in locale: gira sul tuo computer, nessun dato lascia il dispositivo.", "transcribeNow": "Trascrivi ora", "transcribing": "Trascrizione…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/ja-JP/settings.json b/src/i18n/locales/ja-JP/settings.json index 2317b864c..e7e4bb5dc 100644 --- a/src/i18n/locales/ja-JP/settings.json +++ b/src/i18n/locales/ja-JP/settings.json @@ -245,7 +245,7 @@ "help": "タイムライン上のすべてのクリップをまとめた文字起こしです。単語や選択範囲を Backspace / Delete でスキップ(赤色)にできます。赤い部分にカーソルを合わせると元に戻せます。", "noClips": "クリップがまだありません", "noTranscript": "文字起こしがまだありません", - "whisperHint": "文字起こしはローカルの Whisper を使用します。ブラウザ内で実行され、データが端末外に出ることはありません。", + "whisperHint": "文字起こしはローカルの Whisper を使用します。お使いのパソコン上で実行され、データが端末外に出ることはありません。", "transcribeNow": "今すぐ文字起こし", "transcribing": "文字起こし中…", "clipLabel": "クリップ {{index}}", diff --git a/src/i18n/locales/ko-KR/settings.json b/src/i18n/locales/ko-KR/settings.json index 21b064645..4e1d23ca9 100644 --- a/src/i18n/locales/ko-KR/settings.json +++ b/src/i18n/locales/ko-KR/settings.json @@ -245,7 +245,7 @@ "help": "타임라인의 모든 클립을 합친 전사입니다. 단어나 선택 영역에서 Backspace / Delete를 누르면 건너뛴 것으로 표시됩니다(빨간색). 빨간 부분에 마우스를 올리면 복원할 수 있습니다.", "noClips": "아직 클립이 없습니다", "noTranscript": "아직 전사가 없습니다", - "whisperHint": "전사는 로컬 Whisper를 사용합니다. 브라우저에서 실행되며 데이터가 기기를 벗어나지 않습니다.", + "whisperHint": "전사는 로컬 Whisper를 사용합니다. 이 컴퓨터에서 실행되며 데이터가 기기를 벗어나지 않습니다.", "transcribeNow": "지금 전사하기", "transcribing": "전사 중…", "clipLabel": "클립 {{index}}", diff --git a/src/i18n/locales/pt-BR/settings.json b/src/i18n/locales/pt-BR/settings.json index 080b45edf..95f63653e 100644 --- a/src/i18n/locales/pt-BR/settings.json +++ b/src/i18n/locales/pt-BR/settings.json @@ -245,7 +245,7 @@ "help": "Transcrição agregada de todos os clipes da linha do tempo. Backspace / Delete em uma palavra ou seleção marca como ignorada (em vermelho). Passe o mouse sobre um trecho vermelho para restaurá-lo.", "noClips": "Nenhum clipe ainda", "noTranscript": "Nenhuma transcrição ainda", - "whisperHint": "A transcrição usa o Whisper local — roda no seu navegador, nenhum dado sai do dispositivo.", + "whisperHint": "A transcrição usa o Whisper local — roda no seu computador, nenhum dado sai do dispositivo.", "transcribeNow": "Transcrever agora", "transcribing": "Transcrevendo…", "clipLabel": "Clipe {{index}}", diff --git a/src/i18n/locales/ru/settings.json b/src/i18n/locales/ru/settings.json index aa3d0209b..ccec01ca2 100644 --- a/src/i18n/locales/ru/settings.json +++ b/src/i18n/locales/ru/settings.json @@ -245,7 +245,7 @@ "help": "Сводная расшифровка всех клипов на таймлайне. Backspace / Delete по слову или выделению помечает его как пропущенное (красным). Наведите курсор на красный фрагмент, чтобы вернуть его.", "noClips": "Клипов пока нет", "noTranscript": "Расшифровки пока нет", - "whisperHint": "Расшифровка использует локальный Whisper — работает в вашем браузере, данные не покидают устройство.", + "whisperHint": "Расшифровка использует локальный Whisper — работает на вашем компьютере, данные не покидают устройство.", "transcribeNow": "Расшифровать сейчас", "transcribing": "Расшифровка…", "clipLabel": "Клип {{index}}", diff --git a/src/i18n/locales/tr/settings.json b/src/i18n/locales/tr/settings.json index 2f23d40e4..d81852dd4 100644 --- a/src/i18n/locales/tr/settings.json +++ b/src/i18n/locales/tr/settings.json @@ -245,7 +245,7 @@ "help": "Zaman çizelgesindeki tüm kliplerin birleşik dökümü. Bir kelimede veya seçimde Backspace / Delete tuşuna basmak onu atlanmış (kırmızı) olarak işaretler. Kırmızı bölümün üzerine gelerek geri alabilirsiniz.", "noClips": "Henüz klip yok", "noTranscript": "Henüz döküm yok", - "whisperHint": "Döküm yerel Whisper kullanır — tarayıcınızda çalışır, hiçbir veri cihazdan çıkmaz.", + "whisperHint": "Döküm yerel Whisper kullanır — bilgisayarınızda çalışır, hiçbir veri cihazdan çıkmaz.", "transcribeNow": "Şimdi dökümünü çıkar", "transcribing": "Döküm çıkarılıyor…", "clipLabel": "Klip {{index}}", diff --git a/src/i18n/locales/vi/settings.json b/src/i18n/locales/vi/settings.json index f7c47b113..c2fc75d7b 100644 --- a/src/i18n/locales/vi/settings.json +++ b/src/i18n/locales/vi/settings.json @@ -245,7 +245,7 @@ "help": "Bản chép lời gộp của mọi clip trên dòng thời gian. Nhấn Backspace / Delete trên một từ hoặc vùng chọn để đánh dấu là bỏ qua (màu đỏ). Di chuột lên đoạn màu đỏ để khôi phục.", "noClips": "Chưa có clip nào", "noTranscript": "Chưa có bản chép lời", - "whisperHint": "Việc chép lời dùng Whisper cục bộ — chạy ngay trong trình duyệt, không dữ liệu nào rời khỏi thiết bị.", + "whisperHint": "Việc chép lời dùng Whisper cục bộ — chạy ngay trên máy của bạn, không dữ liệu nào rời khỏi thiết bị.", "transcribeNow": "Chép lời ngay", "transcribing": "Đang chép lời…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/zh-CN/settings.json b/src/i18n/locales/zh-CN/settings.json index 536db980e..1fd9f2a83 100644 --- a/src/i18n/locales/zh-CN/settings.json +++ b/src/i18n/locales/zh-CN/settings.json @@ -245,7 +245,7 @@ "help": "时间轴上所有片段的合并转录。对某个词或选区按退格 / Delete 会将其标记为跳过(红色)。将鼠标悬停在红色片段上可恢复。", "noClips": "暂无片段", "noTranscript": "暂无转录", - "whisperHint": "转录使用本地 Whisper — 在您的浏览器中运行,数据不会离开本设备。", + "whisperHint": "转录使用本地 Whisper — 在您的电脑上运行,数据不会离开本设备。", "transcribeNow": "立即转录", "transcribing": "转录中…", "clipLabel": "片段 {{index}}", diff --git a/src/i18n/locales/zh-TW/settings.json b/src/i18n/locales/zh-TW/settings.json index 8483874ac..1210a1db0 100644 --- a/src/i18n/locales/zh-TW/settings.json +++ b/src/i18n/locales/zh-TW/settings.json @@ -246,7 +246,7 @@ "help": "時間軸上所有片段的合併逐字稿。對某個字或選取範圍按 Backspace / Delete 會標記為略過(紅色)。將滑鼠移到紅色片段上即可還原。", "noClips": "尚無片段", "noTranscript": "尚無逐字稿", - "whisperHint": "逐字稿使用本機 Whisper — 在您的瀏覽器中執行,資料不會離開本裝置。", + "whisperHint": "逐字稿使用本機 Whisper — 在您的電腦上執行,資料不會離開本裝置。", "transcribeNow": "立即產生逐字稿", "transcribing": "轉錄中…", "clipLabel": "片段 {{index}}", diff --git a/technical-documentation/architecture/transcription-and-captions.md b/technical-documentation/architecture/transcription-and-captions.md index 1ae45bc0e..c9694ec79 100644 --- a/technical-documentation/architecture/transcription-and-captions.md +++ b/technical-documentation/architecture/transcription-and-captions.md @@ -55,17 +55,32 @@ factors — lives in ### Per-platform backend -| Platform | Helper binary | Runtime device candidates | +The binary is named `whisper-stt-server` (`.exe` on Windows) on **every** +platform — the backend is a build-time flag, not a file name, and +`binaryNameForBackend` +([`electron/stt/gpuDetector.ts:60`](../../electron/stt/gpuDetector.ts:60)) +returns that single name. What varies is which backend was compiled in: + +| Platform | Backend compiled in | Runtime device candidates | |---|---|---| -| macOS arm64 | `whisper-stt-server-metal` | Metal, CPU | -| macOS x64 | `whisper-stt-server-cpu` | CPU | -| Windows x64 | `whisper-stt-server-vulkan` | Vulkan, CPU | -| Linux x64 | `whisper-stt-server-vulkan` | Vulkan, CPU | +| macOS arm64 | Metal (`-DOSC_ENABLE_METAL=ON`) | Metal, CPU | +| macOS x64 | CPU only | CPU | +| Windows x64 | Vulkan (`-DOSC_ENABLE_VULKAN=ON`) | Vulkan, CPU | +| Linux x64 | Vulkan (`-DOSC_ENABLE_VULKAN=ON`) | Vulkan, CPU | whisper.cpp chooses the device in `whisper_backend_init`; the helper reports what actually bound (via `ggml_backend_dev_name()`) and `SttManager` returns it verbatim in the response. +> `ggml_backend_dev_name()` returns a *device* name carrying an index — +> `Vulkan0`, `CUDA0`, `MTL0` — not the backend's title. Metal is the trap: +> ggml-metal builds its device name from `GGML_METAL_NAME` (`"MTL"`), so the +> string `"Metal"` appears nowhere in it. `detect_active_backend()` matched +> `"Metal"` and therefore reported `whispercpp-cpu` on every Apple Silicon run +> while the Metal backend was in fact bound. It now matches `MTL` and filters on +> `ggml_backend_dev_type()` being GPU/IGPU, which also drops ggml-blas +> (Accelerate on macOS, device type ACCEL) from consideration. + ### Word-level alignment (DTW token timestamps) 1. **Decode** — `whisper_full()` runs with the default sampling parameters @@ -170,6 +185,61 @@ curl -X POST -F "file=@test.wav" -F "language=auto" -F "response_format=verbose_ The helper's stderr logs the actual backend it bound (`whispercpp-vulkan` / `-metal` / `-cpu`). +`npm run test:whisper-stt` round-trips a real clip through a real helper and +checks the response against the invariants below — segments and per-word +timings present, the DTW guardrail passed, `backend` reporting GPU offload on a +GPU-capable host, `detected_language` resolved rather than echoed, word times +monotonic and inside the clip, and WER against a reference. On macOS it +synthesizes its own clip with `say`, so it needs no fixture; elsewhere pass +`--wav ` (and optionally `--ref ""`). This is the check +that the unit tests structurally cannot make: they mock `fetch`, so they assert +against a hand-written fixture rather than the binary. + +#### What the staged directory has to contain + +`electron/native/bin/-/` is not just the executable. whisper.cpp is +built as **shared** libraries, so the helper is dynamically linked against +`whisper` plus the ggml backends, and every one of them has to sit next to the +binary: + +- **Windows** — `whisper.dll`, `ggml.dll`, `ggml-base.dll`, `ggml-cpu.dll`, + `ggml-vulkan.dll`. +- **macOS** — `libwhisper.*.dylib`, `libggml{,-base,-cpu,-blas,-metal}.*.dylib`, + plus the version symlink farm (`libggml.dylib` → `libggml.0.dylib` → + `libggml.0.15.1.dylib`). +- **Linux** — the `.so` equivalents, including the `.so.` and + `.so.` links. + +Two macOS/Linux-specific hazards, both of which shipped silently: + +1. **The sidecar glob.** CMake emits `ggml-base.dll` on Windows but + `libggml-base.dylib` / `libggml-base.so.0` elsewhere. A glob written as + `ggml*.*` matches only the Windows spelling, so the staging step copied + nothing but `libwhisper` on macOS and Linux and the helper died in dyld + before `main()`. `build_variant()` now globs the `lib`-prefixed and + `.so.`-suffixed forms too, uses `cp -a` to keep the symlink farm intact + instead of dereferencing each link into a full copy, and **fails the build** + when it stages zero libraries. +2. **Absolute rpaths.** CMake bakes an `LC_RPATH` pointing at its own build + tree, which resolves on the machine that built it and nowhere else — delete + `.cache/`, or download the CI artifact onto a different runner, and the + helper aborts (SIGABRT, exit 134) in the loader. + `relocate_macos_rpaths()` strips every absolute `LC_RPATH` from the + executable and each dylib, adds `@loader_path`, and re-signs ad-hoc + (editing a Mach-O header invalidates its signature, and arm64 refuses to + execute a modified unsigned image). + +`scripts/stage-whisper-stt.sh` gates the installer on actually loading the +binary with a scrubbed `PATH`. That gate was written for Windows, where the +loader dies mute, so it asserted only that the process *printed something* — +and dyld and `ld.so` are chatty, so a macOS helper missing every ggml dylib +printed a four-line loader error and passed. It now requires the +`[whisper-stt] boot:` line that `main()` itself emits, which proves execution +rather than noise, and it no longer depends on `timeout` (GNU coreutils, absent +from a stock macOS runner — unqualified, `env: timeout: No such file or +directory` was itself enough "output" to satisfy the old check without ever +starting the binary). + ## The transcription contract The wire types in `electron/stt/transcriptionContract.ts` are shared across @@ -390,16 +460,31 @@ it deletes data first window and slightly improve WER. Needs a UI control wired through `setTranscript` and a mapping from the UI string to a whisper.cpp language token. + + Note that `"auto"` is a *request* value only. The helper used to echo the + request straight back into `detected_language`, so with no selector the field + was permanently the literal string `"auto"`: the media stage's "detected + language" line ([`src/components/ai-edition/Modals.tsx:1763`](../../src/components/ai-edition/Modals.tsx:1763)) + displayed it verbatim, and `transcribe.ts` wrote it onto + `AxcutTranscript.language`. It now reports `whisper_full_lang_id()` — the + detected language under `"auto"`, the forced one otherwise. - **No CUDA build.** Vulkan already accelerates NVIDIA on Windows and Linux, but a dedicated CUDA build can be added later via `OSC_ENABLE_CUDA=ON`; the build script and CMake already accept the flag, the default matrix doesn't build it yet. - **No CoreML/ANE encoder.** Metal already covers Apple GPU; CoreML is a future perf refinement. -- **HTTP integration test not on CI.** No automated `POST /inference` - round-trip against a known WAV on the macOS-ARM/Metal, - Ubuntu-x64/Vulkan, Windows-x64/Vulkan matrix. The native build - matrix is the right place to land it. +- **HTTP integration test not on CI.** `npm run test:whisper-stt` does the + `POST /inference` round-trip and asserts the contract, but nothing runs it on + the macOS-ARM/Metal, Ubuntu-x64/Vulkan, Windows-x64/Vulkan matrix. The native + build matrix (`build-whisper-stt.yml`) is the right place to land it — each + job already has the freshly built helper on disk, so it only needs a clip and + the model. Off macOS the harness needs `--wav`, so a committed fixture (or a + runner-side TTS) is the missing piece. Until then the three GPU paths are + verified only by whoever runs it locally: **macOS arm64/Metal was verified + this way on 2026-07-30 (WER 0.0000 on an English and a French clip, rtf + 0.18–0.25 on an M1); Vulkan on Windows and Linux has not been re-checked + since the backend-detection fix.** - **No C++ unit tests.** The WAV reader and the DTW-inactive guardrail in `electron/native/whisper-stt/src/main.cpp` are exercised only at runtime. diff --git a/technical-documentation/engineering/build-and-packaging.md b/technical-documentation/engineering/build-and-packaging.md index fb45dc8be..c1e623ed5 100644 --- a/technical-documentation/engineering/build-and-packaging.md +++ b/technical-documentation/engineering/build-and-packaging.md @@ -33,7 +33,8 @@ A usable full package depends on generated artifacts that are not committed: | macOS ScreenCaptureKit capture helper and cursor helper | `electron/native/bin/darwin-arm64/` or `darwin-x64/` | Full Xcode, Swift, SwiftPM; Command Line Tools alone may be insufficient | | Whisper STT server and ggml/whisper backend libraries | `electron/native/bin/-/` | CMake plus host compiler; Metal on Apple Silicon, Vulkan SDK on supported Windows/Linux builds, CPU fallback, optional CUDA | | Native D3D11 compositor addon | `electron/native/compositor-view/build/compositor_view.node` | Rust MSVC toolchain, Visual Studio/Windows SDK, LLVM/libclang, and the exact pinned shared FFmpeg SDK | -| FFmpeg runtime files | matching `electron/native/bin/-/` directory | Downloaded by `fetch:ffmpeg` for the Windows build path | +| Native Metal compositor addon | `electron/native/bin/darwin-/compositor_view.node` (plus a dev copy under `electron/native/compositor-view/build/`) | Rust, Xcode, and the LGPL FFmpeg tree from `fetch:ffmpeg:mac` | +| FFmpeg runtime files | matching `electron/native/bin/-/` directory | Downloaded by `fetch:ffmpeg` on Windows; **built from source** by `fetch:ffmpeg:mac` on macOS (~5 min) — BtbN publishes no macOS target and every circulating macOS build is GPL, which would relicense this MIT app | Electron-builder copies only the matching `electron/native/bin/-/` directory into each package. The compositor `.node` file is included by the Windows `files` rule and unpacked from ASAR because native addons cannot be loaded from inside the archive. @@ -50,13 +51,28 @@ Electron-builder copies only the matching `electron/native/bin/- A stale addon **fails silently rather than erroring**. Scene fields are `#[serde(default)]` on the Rust side, so an addon that predates a contract change does not reject the payload: it ignores the unknown key, takes the default, and falls back to older behaviour. Nothing appears in any log, and the symptom ("the feature does nothing") is indistinguishable from a TypeScript bug. This is not hypothetical — on 2026-07-27 a build shipped an addon three days older than the commit adding `cursorSprites`, custom cursor themes silently rendered as the built-in art, and the resulting investigation blamed the wrong layer entirely. -`scripts/before-pack.cjs` runs as electron-builder's `beforePack` hook and fails the build when the addon is older than `crates/compositor/src/`, `crates/compositor-view-napi/src/`, or either `Cargo.toml`. It only checks when packaging for Windows. The fix it prints is: +`scripts/before-pack.cjs` runs as electron-builder's `beforePack` hook and fails the build when the addon is older than `crates/compositor/src/`, `crates/compositor-view-napi/src/`, or either `Cargo.toml`. The fix it prints is: ```bash -npm run build:native:compositor +npm run build:native:compositor # Windows +npm run build:native:compositor:mac # macOS ``` -CI is unaffected: `.github/workflows/build.yml` already uses `build:win`, which rebuilds before packaging. +**On macOS it also asserts the payload is complete**, which is a stronger check than freshness and exists because the hook used to return early on every non-Windows platform. That gap was not theoretical: the macOS CI job built the ScreenCaptureKit helper but never ran `fetch:ffmpeg:mac` or `build:native:compositor:mac`, so the `.app` it produced had no compositor addon — preview and export dead in the installed app, with nothing in any log. Nothing caught it, because the one guard that would have was Windows-only. + +The hook now reads `electron/native/bin/darwin-/` — the directory `mac.extraResources` ships wholesale, so "present here" means "present in the installed app" — and refuses to package unless it holds all of: + +| Required | Without it | +|---|---| +| `compositor_view.node` | preview and every export render nothing | +| `libavcodec/libavformat/libavutil.*.dylib` | the addon cannot load at all (dyld error at `require()`) | +| `whisper-stt-server` | transcription and captions fail with a developer error shown to end users | +| `libggml*.dylib` | the helper dies in dyld before `main()`; STT times out with no diagnostic | +| `openscreen-screencapturekit-helper` | native screen capture unavailable | + +It then applies the same staleness comparison to the **shipped** addon (the arch-tagged copy), not the dev copy under `electron/native/compositor-view/build/`, since the arch-tagged one is what electron-builder actually packages. + +CI: the Windows job runs `build:win`, which rebuilds before packaging. The macOS job spells its steps out — it needs `--dir` plus a hand-rolled DMG and signing — and had drifted from the `build:mac` recipe; it now vendors the LGPL ffmpeg tree and builds the Metal addon before packing, both cached. The check compares modification times, so `git checkout` (which restamps source files) can occasionally flag an addon that is genuinely fine. That trade is deliberate — a false alarm costs one rebuild, whereas a missed stale addon ships a broken installer. Run `node scripts/before-pack.cjs` on its own to see the verdict without packaging. @@ -70,6 +86,8 @@ The default electron-builder target is NSIS, with an assisted installer that all ### macOS +> **The macOS job is currently disabled** (`if: false` in `build.yml`) because 1.8.0 ships Windows-only. That flag is release-branch-only and must not reach `main` when promoting, or every later release becomes Windows-only too. Until it is lifted, the macOS packaging path — including the compositor and ffmpeg steps described above — is exercised only by `npm run build:mac` locally. + Electron-builder targets DMG for both `arm64` and `x64`, enables hardened runtime, and applies `macos.entitlements` to the app and inherited code. The entitlements allow Electron JIT/native library loading and audio, camera, and screen capture. The configuration itself sets `notarize: false`; release CI packages the `.app`, creates and signs the DMG manually, submits stable tags to `notarytool`, staples the ticket, and validates Gatekeeper. Pre-release tags skip DMG signing/notarization, and missing Apple credentials produce an unsigned artifact. ### Linux and Nix