Skip to content

fix(pipeline): let cancellation reach the processes it is meant to stop - #535

Merged
thcp merged 1 commit into
0.16.1from
fix/519-cancellation-subprocesses
Aug 31, 2026
Merged

fix(pipeline): let cancellation reach the processes it is meant to stop#535
thcp merged 1 commit into
0.16.1from
fix/519-cancellation-subprocesses

Conversation

@thcp

@thcp thcp commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Fixes #519. Independent; branches off 0.16.1.

1. Two ffmpeg calls were unregistered, so cancel could not reach them

runner.py:79 and :130 used subprocess.run(), which cannot be interrupted -- POST /cancel sets the flag but nothing looks at it until the call returns.

Cancelling during "Preparing audio..." on a 400 MB .mp4 was a no-op for up to TIMEOUT_FFMPEG per call, and that path runs both the video extract and the transcode.

Both now go through _run_registered_ffmpeg, mirroring collect._run_ffmpeg which registers for exactly this reason. It deregisters in a finally, so a failure cannot leave a stale entry a later cancel would terminate on the wrong job.

2. Two of three workers never armed the parent-death watchdog

STEMDECK_PARENT_PID was set in exactly one place (separate.py) and read in exactly one place (demucs_worker.py).

  • A Force-Quit during a vocal split orphaned an onnxruntime process holding the GPU, with nobody to collect the result.
  • A section pass outlived the parent whose TIMEOUT_SECTIONS was its only bound.

The watchdog moves to app/core/process.py -- where process_exists already lived specifically for it -- so all three workers share one implementation rather than three copies. All three spawn sites export the pid.

Poll interval preserved at 1.0s. I initially wrote 5s in the shared version, which would have silently slowed the existing demucs watchdog. Caught before commit.

3. A running vocal split was uncancellable by construction

cancel_job returns early for a done job -- and a vocal split only ever runs on a done job, so cancel_requested was never even set, while the split held _pipeline_lock and stalled the whole import queue for its duration.

Cancel now terminates the worker for that specific case. The split's own error path marks it failed and releases the lock, so nothing else was needed.

Verification

New tests/test_cancellation_reach.py, 12 tests. Confirmed not vacuous -- reverting the vocal-split cancel and one parent-pid export fails 2.

Two of them are deliberately structural (test_every_worker_spawn_exports_the_parent_pid, test_every_worker_arms_the_watchdog): they assert across all three workers, so a fourth worker added later without the watchdog fails the suite rather than silently orphaning a process.

ruff check       All checks passed
ruff format      94 files already formatted
pytest tests/    905 passed, 2 failed (pre-existing ogg pair)

Two existing test files updated, not fixed

  • test_worker_parent_watchdog.py targeted demucs_worker._arm_parent_watchdog and demucs_worker.threading; both moved to app.core.process.
  • test_video_status.py stubbed runner_mod.subprocess.run, which the extract no longer calls; it now stubs _run_registered_ffmpeg.

Behaviour is unchanged in both cases -- worth a look to confirm you agree the seams moved sensibly.

Three stages ignored cancel entirely.

The local-upload ffmpeg calls used subprocess.run(), which cannot be
interrupted: POST /cancel sets the flag but nothing looks at it until the call
returns. Cancelling during "Preparing audio..." on a 400 MB .mp4 was a no-op
for up to TIMEOUT_FFMPEG per call, twice over on that path since it runs both
the video extract and the transcode. Both go through _run_registered_ffmpeg
now, mirroring collect._run_ffmpeg, which registers for exactly this reason.

Only demucs_worker armed the parent-death watchdog, and only separate.py
exported STEMDECK_PARENT_PID. A Force-Quit during a vocal split therefore
orphaned an onnxruntime process holding the GPU with nobody to collect the
result, and a section pass outlived the parent whose TIMEOUT_SECTIONS was its
only bound. The watchdog moves to app/core/process.py -- where process_exists
already lived for it -- and all three workers arm it, all three spawn sites
export the pid. Poll interval stays 1.0s, matching what demucs_worker used.

A running vocal split was uncancellable by construction: cancel_job returns
early for a done job, and a split only ever runs on a done job, so the flag was
never even set while the split held _pipeline_lock and stalled the import
queue for its full duration. Cancel now terminates the worker for that case;
the split's own error path marks it failed and releases the lock.

_run_registered_ffmpeg deregisters in a finally, so a failure cannot leave a
stale entry that a later cancel would terminate on the wrong job.

Two existing test files needed updating rather than fixing:
test_worker_parent_watchdog targeted demucs_worker._arm_parent_watchdog, now
app.core.process.arm_parent_watchdog; test_video_status stubbed
subprocess.run, which the extract no longer calls. Behaviour is unchanged in
both.

Refs #519
@thcp
thcp marked this pull request as ready for review August 31, 2026 21:17
@thcp
thcp merged commit 3e89b45 into 0.16.1 Aug 31, 2026
8 checks passed
@thcp
thcp deleted the fix/519-cancellation-subprocesses branch August 31, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant