fix(setup): bound the GPU probe and record which step is running

[thcp]

Fixes #502.

Reported by @raniaamina: Debian Sid with an RTX 5070Ti, first launch sits forever with no progress and nothing in setup.log. Reproduced on 0.15.2 and again on 0.16.1. macOS ARM on the same release is fine.

The hang

verify_cuda_torch used Command::output() with no timeout -- the only subprocess family in main.rs not routed through command_output_with_timeout.

What it runs is not a cheap probe. It deliberately forces a real kernel launch, because torch.cuda.is_available() returns True even for a wheel with no kernels for the device (#217):

import torch
exit(1) if not torch.cuda.is_available() else None
(torch.ones(8, device='cuda') * 2).sum().item()
torch.cuda.synchronize()

That launch is precisely what wedges when the installed wheel and the host driver disagree, and it blocks in the kernel where it cannot be interrupted. Setup then stopped permanently.

Debian Sid plus a Blackwell card is exactly that territory -- the reporter notes the 5070Ti wants a newer CUDA build than what ships. wheel_tag does correctly pick cu128 for compute capability >= 10, so the wheel choice is not the bug; surviving a bad pairing is.

It is now bounded at 120s. A probe that cannot answer is a failed probe, so the existing fallback applies -- the caller restores the CPU wheels and the app starts on CPU rather than not starting at all. The timeout path is logged explicitly, naming a driver/wheel mismatch as the usual cause.

verify_mps_torch gets the same treatment. Nothing has been seen to hang there, but one GPU probe being bounded and the other not was an accident, not a decision.

Why nobody could diagnose it

setup.log only ever recorded failures. No step logged on entry or on success, so a hang wrote nothing at all and the log was indistinguishable from a launch that never happened. That is why the issue sat since 30 August with no way to narrow it down, even after asking the reporter for logs.

Every setup step now logs on entry:

[...] [stemdeck] step: ensure-workspace
[...] [stemdeck] step: ensure-external-assets
[...] [stemdeck] step: gpu-setup          <- last line = where it stopped

Steps run in sequence, so the last line names the step that did not finish. Entry markers alone give that, at one line per step rather than a completion marker on every return path.

Verification

cargo fmt --check   OK
cargo clippy        0 errors
cargo test          70 passed, 0 failed

Two new tests: one drives command_output_with_timeout with a process that never exits and asserts it gives up (~2s, not 300s); one pins the timeout to a sane range.

A caveat worth stating plainly: verify_cuda_torch is behind #[cfg(not(target_os = "macos"))], and neither macOS nor ci.yml compiles it (#531). I type-checked it by temporarily widening just that one gate -- cargo check reported no errors -- then restored all 19 macOS-excluded gates and confirmed the count. It has still never been built for a real Linux target. #531 remains the right fix for that.

What this does and does not do

Does: stop a wedged GPU probe from bricking first launch, and make the next report of this class diagnosable from one log line.

Does not: make the 5070Ti work on CUDA. If the driver and the cu128 wheel genuinely disagree, this converts an indefinite hang into a clean CPU fallback with a logged reason. Getting Blackwell onto CUDA is a separate question -- likely a newer torch than the pinned 2.8.0 -- and worth its own issue once the reporter can tell us what the probe actually said.

Two other silent-stall candidates I did not change, since the logging will now identify them if they are the real cause: the Linux FFmpeg download (download_file, 30-minute timeout, no progress events, single non-CDN host) and the CUDA torch install (~3 GB with stdout null'd, 20-minute cap).

---

Added after the first round of testing

@raniaamina tried the branch and hit two things it did not cover. Both are now
fixed here.

The CUDA install was still completely silent (#502)

She waited fifteen minutes, saw nothing, and closed the app. That was the right
read of what was on screen: run_pip_install set .stdout(Stdio::null()),
passed --quiet, and read through child_output_with_timeout, which collects
to EOF. Nothing reached the log or the screen until the process exited. Two
passes on Linux at twenty minutes each. The setup screen had only canned text on
a timer, because this step emitted no progress events at all.

The download is not small: the cu128 torch wheel for Linux is 847 MB and
nvidia-cublas alone is 566 MB, before the dozen other nvidia-* packages.

That silence is what produced her second report. Pass 1 installs CUDA torch with
--no-deps and pass 2 installs the runtime it dlopens, so stopping between them
leaves torch unable to import at all, which is the ImportError: libcudnn.so.9
she posted. It self-heals on the next launch, silently, so she killed it again.

Now streamed. --progress-bar raw is pip's machine-readable mode, which is what
it emits when stdout is a pipe rather than a terminal. Lines are parsed and go
to setup.log (readable lines only) and to a new setup-progress event that
the setup screen renders, so she sees
Downloading nvidia_cublas_cu12-... 210 / 566 MB instead of a spinner.

--progress-bar raw has only existed since pip 24.1 and the packaging script
pins no pip version, so a pip that will not take the flag is detected from its
argument-parse error and the pass is retried without it.

The subprocess timeout was not a bound (#583)

This is what the Linux Rust Check has been failing on since 8dc1577.

The timeout branch killed the child and then joined both reader threads, on the
reasoning that killing the child closes its pipes. It closes only the ends the
child itself held. A grandchild inherited the same write ends, so the pipe stays
open and the join waits for the grandchild.

The test's stand-in makes it concrete: sh -c "sleep 300" forks on Ubuntu
rather than execs, so killing sh leaves sleep holding the stderr pipe for its
full 300 seconds. Against a 2 second deadline the call took 300.05. Readers are
now detached on the timeout path; the success path still joins, because there
the child has exited and the collected output is the return value.

This is #502 one level down, and run_pip_install uses the same path with a
20 minute timeout.

Also

cuda_index_url was orphaned by the candidate loop two commits ago and was
failing cargo clippy -- -D warnings as dead code, so this branch's CI was red
before any of the above. Removed; wheel_tag was only reachable through it and
is now marked test-only.

Rebased onto main, which it predated.

Gate

Linux: 95 Rust tests, the wedged-probe case in 2.01s rather than 300. Windows:
88. clippy with and without --tests, and fmt, clean on both. Playwright 127,
tests/js, node --check on static/js and desktop/ui. No Python touched.

Closes #583

[thcp]

Update: this now aims to make the GPU work, not just fail cleanly

The first commit bounded the hang. On its own that left the 5070Ti on CPU, which misses the point. 8dc1577 addresses the wheel selection itself.

What was wrong

wheel_tag handed every Blackwell card cu128 and nothing else, regardless of driver:

if major >= 10 { return "cu128"; }

So on Debian Sid with a CUDA 13 driver, cu128 is the only build it was ever offered. When that pairing does not work there is no GPU and no explanation. @raniaamina's own finding was that the card wants a newer CUDA build than what ships.

Separately, cuda_tag knew CUDA 11 and 12 only. A 13.x driver fell into the catch-all and was handed cu124 -- two major versions behind, on every card, not just Blackwell.

What it does now

Wheel selection returns an ordered list, because the right wheel for a card/driver pairing is not reliably knowable from here:

GPU	driver	tries
Blackwell (cap >= 10)	CUDA 13+	cu130 (torch 2.11.0), then cu128 (torch 2.8.0)
Blackwell	CUDA 12	cu128 only -- no point offering a build that postdates the driver
anything else	any	driver heuristic, as before

ensure_torch_device walks that list: install, verify, move on only when a build fails. CPU is the answer after every candidate fails, not after the first. Each attempt logs its tag and torch version, so a failure now records which builds were tried and how each failed.

Verified against the live PyTorch index: cu130 carries torch up to 2.14.0 but torchaudio only to 2.11.0, so 2.11.0 is the newest pairing where both exist. 2.8.0+cu128 is still published, so the existing path is unaffected.

A newer torchaudio is safe here for the same reason the existing 2.8.0 override is: demucs' torchaudio.save() dispatches through soundfile, a hard dependency, not torchaudio's own codecs.

Verification

cargo fmt --check   OK
cargo clippy        0 errors
cargo test          74 passed, 0 failed

The wheel logic is pure, so it is now compiled in test builds on every platform rather than only off-macOS. Its four tests could not run on this machine before; they run everywhere now, which is rather the point of testing a decision table.

The cfg-widening check earned its keep. It caught [13.., _] -- an unstable slice pattern (exclusive_range_pattern) that compiles on no Rust release, and that no check available on macOS would have reported. That would have failed the Linux release build. All 21 gates restored and counted afterwards. This is a concrete argument for #531.

What I still cannot promise

I have no Blackwell GPU and no Linux machine. This is not verified against the actual hardware -- I can only say the logic is right, the wheels exist, and the fallback means a wrong first guess costs time rather than the GPU.

Worth asking @raniaamina to retry and send setup.log. It will now name the step, the tags tried, and how each failed -- which is the information nobody had.

If cu130/torch 2.11.0 also fails to verify, the log will say so, and the next step is a torch version question rather than another guess.