fix(pipeline): let cancellation reach the processes it is meant to stop - #535
Merged
Conversation
Three stages ignored cancel entirely. The local-upload ffmpeg calls used subprocess.run(), which cannot be interrupted: POST /cancel sets the flag but nothing looks at it until the call returns. Cancelling during "Preparing audio..." on a 400 MB .mp4 was a no-op for up to TIMEOUT_FFMPEG per call, twice over on that path since it runs both the video extract and the transcode. Both go through _run_registered_ffmpeg now, mirroring collect._run_ffmpeg, which registers for exactly this reason. Only demucs_worker armed the parent-death watchdog, and only separate.py exported STEMDECK_PARENT_PID. A Force-Quit during a vocal split therefore orphaned an onnxruntime process holding the GPU with nobody to collect the result, and a section pass outlived the parent whose TIMEOUT_SECTIONS was its only bound. The watchdog moves to app/core/process.py -- where process_exists already lived for it -- and all three workers arm it, all three spawn sites export the pid. Poll interval stays 1.0s, matching what demucs_worker used. A running vocal split was uncancellable by construction: cancel_job returns early for a done job, and a split only ever runs on a done job, so the flag was never even set while the split held _pipeline_lock and stalled the import queue for its full duration. Cancel now terminates the worker for that case; the split's own error path marks it failed and releases the lock. _run_registered_ffmpeg deregisters in a finally, so a failure cannot leave a stale entry that a later cancel would terminate on the wrong job. Two existing test files needed updating rather than fixing: test_worker_parent_watchdog targeted demucs_worker._arm_parent_watchdog, now app.core.process.arm_parent_watchdog; test_video_status stubbed subprocess.run, which the extract no longer calls. Behaviour is unchanged in both. Refs #519
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #519. Independent; branches off
0.16.1.1. Two ffmpeg calls were unregistered, so cancel could not reach them
runner.py:79and:130usedsubprocess.run(), which cannot be interrupted --POST /cancelsets the flag but nothing looks at it until the call returns.Cancelling during "Preparing audio..." on a 400 MB
.mp4was a no-op for up toTIMEOUT_FFMPEGper call, and that path runs both the video extract and the transcode.Both now go through
_run_registered_ffmpeg, mirroringcollect._run_ffmpegwhich registers for exactly this reason. It deregisters in afinally, so a failure cannot leave a stale entry a later cancel would terminate on the wrong job.2. Two of three workers never armed the parent-death watchdog
STEMDECK_PARENT_PIDwas set in exactly one place (separate.py) and read in exactly one place (demucs_worker.py).TIMEOUT_SECTIONSwas its only bound.The watchdog moves to
app/core/process.py-- whereprocess_existsalready lived specifically for it -- so all three workers share one implementation rather than three copies. All three spawn sites export the pid.Poll interval preserved at 1.0s. I initially wrote 5s in the shared version, which would have silently slowed the existing demucs watchdog. Caught before commit.
3. A running vocal split was uncancellable by construction
cancel_jobreturns early for adonejob -- and a vocal split only ever runs on a done job, socancel_requestedwas never even set, while the split held_pipeline_lockand stalled the whole import queue for its duration.Cancel now terminates the worker for that specific case. The split's own error path marks it failed and releases the lock, so nothing else was needed.
Verification
New
tests/test_cancellation_reach.py, 12 tests. Confirmed not vacuous -- reverting the vocal-split cancel and one parent-pid export fails 2.Two of them are deliberately structural (
test_every_worker_spawn_exports_the_parent_pid,test_every_worker_arms_the_watchdog): they assert across all three workers, so a fourth worker added later without the watchdog fails the suite rather than silently orphaning a process.Two existing test files updated, not fixed
test_worker_parent_watchdog.pytargeteddemucs_worker._arm_parent_watchdoganddemucs_worker.threading; both moved toapp.core.process.test_video_status.pystubbedrunner_mod.subprocess.run, which the extract no longer calls; it now stubs_run_registered_ffmpeg.Behaviour is unchanged in both cases -- worth a look to confirm you agree the seams moved sensibly.