fix(api): bound request input that could exhaust memory or stall the loop - #533
Merged
Conversation
…loop Four related holes, all reachable with one unauthenticated request. The trim range had no ceiling. `end` is a float that reaches np.zeros(int(round(duration * sample_rate))) in the click renderer, so ?start=0&end=20000&count_in=1 asked for roughly 7 GB, and another 7 GB in the int16 conversion. A larger value raised MemoryError inside a blanket except, which silently shipped the export with no click rather than failing. _validate_trim_range bounds it by the job's own recorded duration, with a six-hour backstop for a job whose duration was never recorded, and a second of slack because ffprobe's duration can sit a hair under the decoded length. The click render ran synchronously inside async handlers, so all of that allocation and a Python loop over every beat blocked the event loop -- every SSE progress stream and the queue worker stalled behind it. It goes through asyncio.to_thread now. The buffer is float32 rather than float64: the output is 16-bit PCM, so the extra mantissa was never audible and a long export was allocating twice what it needed. The click cache pruned without keep=, so a render larger than the cache budget evicted itself the instant it was written and ffmpeg was handed a missing -i. That is the #482 bug, unfixed on this path. The mixdown write had the same gap against a concurrent render's prune. The body-size guard was scoped to paths ending /sections or /beats, leaving /api/search, /api/playlist, /api/settings and the JSON branch of /api/jobs uncapped -- Starlette buffers the whole body, then json.loads runs it on the event loop. A 200 MB body to /api/search stalled every other request with no valid job or prior state needed. It now applies by method, exempting multipart uploads, which stream to disk under their own 400 MB limit. A chunked request with no Content-Length used to skip the check entirely and fall through to an unbounded request.body(); it gets a 411 now. Six of the eight new tests fail against the old code. Refs #512
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #512. Independent; branches off
0.16.1.Four related holes, each reachable with a single unauthenticated request.
1. The trim range had no ceiling
end: float | None = Query(default=None, gt=0)-- nole=, and the only cross-check wasstart >= end. It reaches:?start=0&end=20000&count_in=1asks for ~7 GB, plus another ~7 GB in the int16 conversion. A larger value raisesMemoryErrorinside a blanketexcept, which silently shipped the export with no click rather than failing._validate_trim_rangebounds it by the job's ownduration_sec, with a six-hour backstop for a job whose duration was never recorded, and 1s of slack -- ffprobe's duration can sit a hair under the decoded length and the UI legitimately asks for the very end of a track. Wired into all three handlers that accept a range.2. The render blocked the event loop
_click_lanewas a plaindefcalled insideasync def. All of the above allocation plus a Python loop over every beat ran on the loop, so every SSE progress stream and the queue worker stalled behind it. Nowasyncio.to_thread.Buffer switched to float32: output is 16-bit PCM, so the extra mantissa was never audible, and a long export was allocating twice what it needed.
3. The click cache evicted its own render
_prune_mixdown_cache(_CLICK_CACHE_DIR)was called withoutkeep=. A render larger than the 500 MB budget evicted itself the instant it was written, and ffmpeg was handed a missing-i. That is the #482 bug, unfixed on this path. The mixdown write had the same gap against a concurrent render's prune; both now passkeep=.4. The body guard covered two paths and had a chunked bypass
Scoped by path suffix (
/sections,/beats), leaving/api/search,/api/playlist,/api/playlist/preview,/api/settingsand the JSON branch of/api/jobsuncapped. A 200 MB body to/api/searchstalled every other request.Now applied by method, exempting multipart uploads (they stream to disk under their own 400 MB limit). A chunked request with no
Content-LengthmadedeclaredNoneand skipped the check entirely -- it gets a 411 now rather than falling through to an unboundedrequest.body().Verification
New
tests/test_request_body_limits.py, 8 tests. Confirmed not vacuous -- reverting the guard scope and the trim bound fails 6 of 8.Two review notes
_EDITOR_BODY_LIMITwas renamed to_JSON_BODY_LIMITsince it is no longer editor-specific.tests/test_jobs_api.pyimported it directly and was updated.411 is an unusual status to return. The alternative was streaming and counting the body ourselves, which is more code in a middleware that runs on every request. Given nothing in the app sends chunked JSON, refusing it outright seemed proportionate -- but if any client does, this would break it.