fix(macos): stop shipping AppleDouble files that break song sections, and give warmup the bundled FFmpeg by thcp · Pull Request #506 · stemdeckapp/stemdeck

thcp · 2026-08-30T21:46:07Z

Fixes #505.

Two macOS desktop defects, both surfacing as warmup failures in setup.log. The first one is not really a warmup problem.

1. AppleDouble sidecars break automatic song sections

make-runtime-pack.sh copies Python with ditto, which preserves extended attributes on every file, then tars the staging tree without disabling copyfile. macOS tar serializes those xattrs as AppleDouble ._name members, and the Rust tar crate that unpacks the pack on the user's machine has no AppleDouble support, so it writes them out as literal files. A 0.16.0 install carries 30,239 of them.

One lands in matplotlib's style directory. matplotlib globs *.mplstyle at import time, matches ._seaborn-v0_8-bright.mplstyle, and dies on byte 45 of the AppleDouble header:

matplotlib/style/__init__.py:209  styles[path.stem] = rc_params_from_file(path, ...)
matplotlib/__init__.py:902        for line_no, line in enumerate(fd, 1)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 45: invalid start byte

matplotlib logs Cannot decode configuration file ... as utf-8. and re-raises, so import matplotlib.pyplot fails, which fails allin1_infer/__init__.py -> .analyze -> .visualize, which fails every importer of allin1_infer.

That includes app/pipeline/section_worker.py. Automatic song section detection was broken outright on macOS 0.16.0, not merely un-prewarmed. Linux and Docker are unaffected: this is a macOS archive artifact.

Fixed on both ends:

Build: strip xattrs and any sidecars from the staging tree, and set COPYFILE_DISABLE=1 before tarring.
Runtime: extract_tar_archive skips ._ entries, so the packs already published are handled without waiting for a rebuild.

The pack script also verifies the finished archive and fails the build if a sidecar survives. That check deliberately does not use tar -tf: macOS tar folds ._name members back into their sibling's metadata while listing exactly as it does while creating, so it reports a clean archive whether or not one is clean. It streams members through Python's tarfile instead, which has no AppleDouble handling.

2. Warmup did not put the bundled FFmpeg on PATH

warmup_models built its Python command without the PATH block start_backend has, so the FFmpeg we downloaded into the data directory was invisible to it. A Finder-launched .app inherits a bare /usr/bin:/bin:/usr/sbin:/sbin, and audio_separator probes for ffmpeg before loading anything:

WARMUP_FAILED vocal_split [Errno 2] No such file or directory: 'ffmpeg'

Smaller blast radius: the real backend does get the right PATH, so the on-demand karaoke split still worked, it just paid the model download mid-job. Recurring on every launch in setup.log since timestamp 1787336901.

The two spawn sites had drifted because each carried its own copy of the PATH logic, so it is extracted into apply_ffmpeg_path and called from both.

Verification

Reproduced both failures against the real installed 0.16.0 runtime, the second under env -i PATH=/usr/bin:/bin:/usr/sbin:/sbin to mimic a Finder launch.
Deleting the sidecars from the installed runtime makes allin1_infer and the full section_worker import set succeed; adding the FFmpeg dir to PATH makes Separator construct.
New regression test extract_tar_archive_drops_apple_double_sidecars builds a .tar.zst containing a sidecar and asserts it is dropped while the real file is kept. Confirmed it fails without the fix.
cargo test 58 passed, cargo fmt --check and cargo clippy clean.
Exercised the pack script's strip-and-verify logic on a synthetic staging tree in both directions: clean archive passes, archive with nested and top-level sidecars is caught.

Note for the release

The extract_tar_archive guard only runs on a fresh runtime install. Anyone already on 0.16.0 keeps their 30k sidecars until the runtime is re-fetched, so this needs a runtime version bump to reach existing macOS installs.

The macOS runtime pack copies Python with `ditto`, which preserves extended attributes on every file, and then tarred the staging tree without disabling copyfile. macOS tar serialized those xattrs as AppleDouble "._name" members, and the Rust tar crate that unpacks the pack on the user's machine has no AppleDouble support, so it wrote 30,239 of them out as literal files. One lands in matplotlib's style directory. matplotlib globs "*.mplstyle" at import time, matches "._seaborn-v0_8-bright.mplstyle", and dies on byte 45 of the AppleDouble header: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 45 That kills `import matplotlib.pyplot`, and with it allin1_infer and every importer of it. Automatic song sections were broken outright on macOS, not merely un-prewarmed: section_worker.py takes the same import path warmup does. Linux and Docker are unaffected. Fixed on both ends. The pack script strips xattrs and sidecars from the staging tree and sets COPYFILE_DISABLE, and extract_tar_archive skips "._" entries so packs already published are handled too. The pack script also verifies the finished archive, because whether macOS tar emits AppleDouble members at all varies by OS version, so none of the guards can be trusted blind. That check cannot use `tar -tf`: macOS tar folds "._name" members back into their sibling's metadata while listing exactly as it does while creating, and reports a clean archive either way. It streams members through Python's tarfile instead, which has no AppleDouble handling. Refs #505

`warmup_models` spawned its Python child without the PATH block `start_backend` has, so the bundled FFmpeg in the data directory was invisible to it. A Finder-launched .app inherits a bare /usr/bin:/bin:/usr/sbin:/sbin, and audio_separator probes for `ffmpeg` before it will load anything, so the karaoke vocal-split model failed every setup run with: WARMUP_FAILED vocal_split [Errno 2] No such file or directory: 'ffmpeg' Smaller blast radius than the sections failure: the real backend does get the right PATH, so the on-demand vocal split still worked, it just paid the model download mid-job instead of during setup. The two spawn sites had drifted because each carried its own copy of the PATH logic, so extract it into apply_ffmpeg_path and call it from both. Refs #505

The template tracks the latest published pre-release, which is now 0.16.0. Unraid users pull exactly this tag, so leaving it at 0.15.2 means their installs silently lag behind. Only the <Repository> tag changes; the Docker image itself is unaffected by the macOS fixes in this branch.

…tor (#523) Fixes #508. Follow-up to #506 -- a regression introduced by that PR. #506 replaced `Archive::unpack` with a per-entry `unpack_in` loop so AppleDouble `._` members could be skipped. The loop reproduced the iteration but not the two things `unpack` does around it. ## 1. Directory deferral `Archive::_unpack` (tar-0.4.45 `archive.rs:245-265`): ```rust // Delay any directory entries until the end (they will be created if needed by // descendants), to ensure that directory permissions do not interfere with // descendant extraction. directories.sort_by(|a, b| b.path_bytes().cmp(&a.path_bytes())); ``` A directory carries its own mode. Created inline in archive order, a `0o555` member exists before its contents are written, and the next file inside it fails `File::create` with EACCES -- `unpack_without_apple_double` returns `Err` and first-run setup dies with no fallback. `ditto` and `tar` both record such modes faithfully, so this was reachable, not theoretical. Deferring costs nothing on a streaming archive: directory entries carry no data, so the second pass applies metadata only. ## 2. `destination` canonicalization Restored before the loop. On Windows this supplies the `\\?\` extended-length prefix so member paths over 260 characters still extract. ## Not affected: traversal protection `unpack_in` rejects `ParentDir` components, strips `RootDir`/`Prefix`, and calls `validate_inside_dst` -- which canonicalizes on every entry -- so zip-slip, absolute members and symlink escapes were blocked throughout. This was a robustness regression, not a security one. ## Impact Latent on shipped artifacts. The published 0.16.0 pack has no restrictive directory members: extracting it end to end produced 27,797 files and 0 sidecars. Windows takes the zip path for updates, so the long-path loss was theoretical there. Fixed because the next pack is not guaranteed to be as forgiving. ## Verification New test `extract_tar_archive_survives_a_read_only_directory_member` builds a `.tar.zst` whose `0o555` directory member precedes its file member -- the order that broke -- and asserts the file extracts and the directory keeps its archived mode. Confirmed the test is not vacuous: with directories created inline it **fails**; with deferral it **passes**. ``` cargo fmt --check OK cargo clippy 0 errors cargo test 59 passed, 0 failed ```

…multitrack (#536) engineMode() returns "chunked" unless the user has set the audioEngine flag to "0", and on that path audioEngine owns the clock while the multitrack is mounted with url: null for visuals only. transport.js handled this everywhere via `audioEngine ?? multitrack`; main.js imported only `multitrack` and called it bare. So on the default configuration: - The footer scrub bar did nothing. It is a full-size cursor: pointer overlay, and clicking or dragging anywhere on it moved neither the playhead nor the audio. - [ and ] seeked the silent multitrack, so nothing happened. - I and O read multitrack.getCurrentTime(), which is pinned at 0 there, so "set loop in at playhead" always wrote 0 no matter where the playhead was, and "set loop out" always wrote max(0, loopStart + 0.5). Space and ruler clicks were unaffected because they live in transport.js. Rather than patching five call sites, transport.js exports the accessor it was already using internally, plus setPlayheadTime. Seeking now goes through setPlayheadTime, which also updates the playhead marker, footer times and presence playhead -- none of which the old multitrack.setTime call did, so the scrub bar would have left the marker stale even had it worked. Same anti-drift shape as apply_ffmpeg_path in #506. The keydown guard excluded HTMLInputElement but not HTMLTextAreaElement, so Space in the settings log viewer started playback instead of scrolling. Fixed alongside, since the guard was being edited anyway. The test is structural rather than DOM-driven: the bug was not a wrong value from a function, it was reaching for the wrong object, so it asserts main.js holds no bare multitrack playback calls. All 8 checks fail against the previous code. Refs #515

…om the bin, and 14 more fixes (#541) Ships the 0.16.1 fix set to `main`. Eighteen merged changes: sixteen fixes from a pre-release bug scan, one refactor, and one feature. ## Why this release exists #506 fixed the macOS AppleDouble bug but is unreleased, so every macOS 0.16.0 user still has a broken runtime: `import matplotlib.pyplot` fails, which kills `allin1_infer`, which kills automatic song sections. Overwriting the 0.16.0 assets would not reach them, the runtime reinstall is gated on a version-string comparison (`desktop/ui/setup.js`), so a new version is the only route. ## What is in it **Data loss and user-visible bugs** - **#509** `settings.json` was written with `write_text` (truncate, then write), and `_load` could not tell a torn file from a first run. A real user lost `port` and `allow_network` from both the file and its mirror. Now atomic, and a corrupt file is preserved as `settings.json.corrupt-<ts>` and recovered from the mirror. - **#521** Deleted songs came back. `reset_all` swallowed per-directory failures, `/api/reset` reported unconditional success, the frontend then wiped its own tombstone, and `restore()` re-adopted every surviving directory on the next start. A server-side deletion record closes it. - **#542** Trashed songs came back too, by a different route. `addTrackToLibrary` deduplicates by source URL, and when the match was in the Trash it deleted the catalog entry without deleting the job. The directory and its registry record outlived their only reference, and `syncWithServer` re-adopted the orphan on the next launch. Any second job sharing the URL was enough to trigger it. Found while testing this release on Windows. - **#515** The footer scrub bar did nothing, and "set loop in at playhead" always wrote 0, `main.js` drove the silent multitrack while `audioEngine` owned the clock. - **#520** A cancel landing between the queue worker's pop and claim stranded a job at `queued` forever: invisible, still counted against capacity, source file never freed, re-queued on every restart. Also, a malformed `registry.json` raised an uncaught `AttributeError` at import and the backend never started. **Security** - **#510** The in-app updater installed an executable from a WebView-supplied URL with no host allowlist, checked against a SHA from the same caller. - **#511** Fork PRs executed arbitrary code on the self-hosted runners, the same machine that builds and signs releases. - **#518** Linux FFmpeg was downloaded, chmod +x and executed with no integrity check at all. - **#517** Deno pulled from `releases/latest` unpinned and unverified into every image; releases could publish without updater assets and stay green; a failed CPU-torch install was silently ignored, shipping a non-CPU torch in the CPU zip. **Robustness** - **#508** A regression in #506 itself: the per-entry unpack loop lost `Archive::unpack`'s directory deferral, so a read-only directory member would fail extraction outright. - **#512** `end` had no upper bound, reaching a multi-GB `np.zeros` on the event loop; the body-size guard covered two paths and was bypassed by chunked encoding. - **#513** SSE slots leaked permanently when a client disconnected before the body started, 200 of those and every progress stream 503s with nothing connected. - **#514** Worker teardown sat outside the `finally`, so an exception left a poisoned CUDA worker warm; cancel was dropped before the CPU fallback, costing 10+ minutes. - **#516** `child_output_with_timeout` never drained child pipes until exit, deadlocking any chatty child. - **#519** Cancellation never reached several pipeline subprocesses, and two of three workers never armed the parent-PID watchdog, so a Force-Quit orphaned a GPU-holding process. **Presentation** - **#543** Three of the eight logos listed in the We Recommend dialog had no file behind them. Analog4Lyfe, Empress Effects and Thomann showed a broken card on every install, with a 404 in the backend log each time the dialog opened. - **#544** That dialog was one flat list of twelve entries with no order a reader could perceive. It is grouped into five categories now, `r/bass` is added so the app matches the README, and the twelve descriptions move out of hardcoded English into the i18n layer across all ten language tables. **Feature** - **#538** Loop regions can be adjusted rather than redrawn, drag either edge independently, or drag the region to slide it. From discussion #507. ## Verification ``` ruff check All checks passed ruff format 101 files already formatted pytest tests/ 947 passed, 2 failed npm run test:js 11/11 playwright 85 passed i18n audit clean cargo fmt clean cargo clippy 0 errors cargo test 60 passed, 1 failed ``` Every fix was individually confirmed present on this branch by grepping for its introduced symbol, rather than trusting merged state, which is how #527 was caught having merged into an orphaned branch instead of the release branch (recovered as #540). **The 3 failures are all pre-existing and reproduce on `main`:** - `test_all_stems_zip_ogg` and `test_ogg_is_still_streamed`, most likely a local ffmpeg built without libvorbis. Unconfirmed, not filed. - `a_free_port_is_granted_as_asked`, a known parallel-execution flake. It probes port 21000 with `std::net::TcpListener` (which sets `SO_REUSEADDR`) then asserts `claim_port` (socket2, without it) binds the same port. Not equivalent, and there is a TOCTOU gap. ## Tested on a real Windows build The branch was packaged with `make-portable.ps1 -CpuOnly` and driven by hand. What that covered: - **#509** port and `allow_network` survived a quit and relaunch, in both the portable file and the AppData mirror. - **#521** a hard-deleted job stayed deleted across a restart. The deletion record self-pruned once the directory was gone, which is the designed behaviour. - **#542** a trashed job stayed in the Trash across a restart, and again across a fresh import of the same URL. Both cases fail on 0.16.0. - **#520** a cancel while queued removed the job and freed its capacity slot, with nothing stranded. - **#519 / #514** a cancel mid-separation wiped the partial output and the queued job started immediately, with no leftover worker process. A quit mid-separation left zero orphaned `python.exe`, and the interrupted job resumed once with `resume_attempts: 1`. ## Two things reviewers should know before tagging **`make-portable.ps1` now has a parser pass and a real run.** It parses clean under Windows PowerShell 5.1, and the full CPU-only Windows package built end to end from this branch, exit code 0. The earlier caveat here is resolved. **Nothing compiles the Linux Rust shell until release time** (#531). `ci.yml` never invokes cargo; Linux Rust is built only by `linux-release.yml`. #518 ships a change living entirely inside `#[cfg(all(unix, not(target_os = "macos")))]`, which could only be type-checked by temporarily widening the cfg gate. If the release build fails, look there first. **#510 is not manually testable.** The update check runs automatically at startup and only surfaces when GitHub has a newer non-prerelease release, so the host allowlist in the installer is not reachable by hand. It exercises itself at release time. ## Not included The Unraid template still pins `0.16.0`, deliberately left for a separate decision.

stemdeckapp#506 replaced Archive::unpack with a per-entry unpack_in loop so AppleDouble "._" members could be skipped. That loop reproduced the iteration but not the two things unpack does around it. Directories are now applied last, reverse-sorted by path, the way Archive::_unpack does (tar-rs#242): a directory carries its own mode, so creating it inline in archive order means a 0o555 member exists before its contents are written and the next file inside it fails with EACCES. On a runtime pack that contains one, first-run setup dies with no fallback. `ditto` and `tar` both record such modes faithfully, so this was reachable. `destination` is canonicalized before the loop again, which on Windows supplies the \\?\ prefix so member paths over 260 characters still extract. Traversal protection was never affected and is unchanged: unpack_in rejects ParentDir components, strips RootDir/Prefix, and canonicalizes against dst on every entry, so zip-slip, absolute members and symlink escapes stay blocked. This was a robustness regression, not a security one. Deferring costs nothing on a streaming archive: directory entries carry no data, so only their metadata is applied in the second pass. The new test builds a .tar.zst whose directory member is 0o555 and whose file member follows it, which is the order that broke. Verified it fails against inline creation and passes with the deferral. Refs stemdeckapp#508

thcp added 3 commits August 30, 2026 22:45

thcp marked this pull request as ready for review August 30, 2026 21:58

thcp merged commit b616392 into main Aug 30, 2026
12 of 13 checks passed

thcp deleted the fix/macos-appledouble-runtime-pack branch August 30, 2026 22:10

thcp mentioned this pull request Aug 31, 2026

0.16.1: macOS song sections, settings data loss, songs coming back from the bin, and 14 more fixes #541

Merged

Navigation Menu

Uh oh!

fix(macos): stop shipping AppleDouble files that break song sections, and give warmup the bundled FFmpeg - #506

fix(macos): stop shipping AppleDouble files that break song sections, and give warmup the bundled FFmpeg#506
thcp merged 3 commits into
mainfrom
fix/macos-appledouble-runtime-pack

thcp commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

thcp commented Aug 30, 2026

1. AppleDouble sidecars break automatic song sections

2. Warmup did not put the bundled FFmpeg on PATH

Verification

Note for the release

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant