Skip to content

Stop the pitch worklet burning the audio thread at zero transpose and at high output rates - #577

Open
thcp wants to merge 2 commits into
mainfrom
fix/idle-pitch-chains
Open

Stop the pitch worklet burning the audio thread at zero transpose and at high output rates#577
thcp wants to merge 2 commits into
mainfrom
fix/idle-pitch-chains

Conversation

@thcp

@thcp thcp commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Two fixes for the same symptom, from opposite directions. #576 and #575 are one
bug; #578 is what is left over once that bug is gone.

Credit

@pywkt found this, and the first commit is his work. He traced the pops on
his laptop to the worklet's input contract, measured the cost of the idle chains
against the real processor, and wrote the fix on a branch. He could not open a
pull request because this repo restricts them to collaborators, so he posted the
branch in #576 and asked me to pull it directly.

The engine code shipped here is his as written: the per-bus lane count, the
_laneArriving and _laneLeft pair, the deferred wiring when the worklet
resolves, and the reordering inside swap() so a bus is live before a lane
reaches it. So are all twelve of the new routing checks. I reviewed it,
reproduced his measurements independently, and changed nothing about the
approach, because there was nothing to change. He is a co-author on the commit.

What I added to his commit is one thing. FakeNode.disconnect in the routing
test ignored its arguments and cleared every connection on the node, so his new
checks would have passed a selective disconnect and a blanket one alike. It now
matches the real overloads and throws on a connection that is not there, which
is what lets those checks actually fail.

@wastedbits reported #575 from the other end. The detail that mattered was
the ladder: fine at 48 kHz, distorted at 88.2 and 96, silent at 176.4 and 192.
A bug that is absent, then audible, then total, along a single axis is a bug
whose cost scales with that axis, and that is what sent me to measure the pitch
chains per sample rate rather than per stem. Without those three data points I
would have read #576 as a slow laptop. It also led straight to the second commit
here, which is a different problem that only shows up at the top of the same
ladder. He then ran pywkt's changes on his own 24-bit/192 kHz Windows machine
and confirmed them, which is the only test of this on real high rate hardware
that exists.


1. Twelve idle pitch chains at zero transpose (#576, #575)

Both engines wired all thirteen semitone buses into the SoundTouch worklet the
moment it loaded. The processor decides whether a semitone is in use by whether
its input has channels, and only takes its bypass path when none do. The comment
there says an unconnected input arrives as an empty array, and the unit test
hands unlisted inputs [] by construction, so nothing ever contradicted it.

Chrome does not do that. A bus wired with nothing playing into it arrives as one
channel of silence. With audio on the unpitched bus only, the processor is handed
[1,1,1,1,1,1,2,1,1,1,1,1,1]. So all twelve pitch inputs read as live, twelve
PitchChains are built on the first render quantum, and each runs WSOLA, the
8th-order anti-alias cascade and the resampler on silence for the whole track.
The bypass never engages, and a chain is only dropped after 256 blocks of an
empty input, which never arrives.

The WSOLA windows are in milliseconds, so the wasted work per second of audio
scales with the context's rate. Offline render, seven stems, no transpose,
headless Chromium, share of real time:

Output rate Only the unpitched bus wired All thirteen wired (main)
44.1 kHz 0.2% 17.8%
48 kHz 0.3% 20.9%
96 kHz 0.8% 78.4%
192 kHz 1.5% over real time

That is #576 and #575 as one bug seen from two ends of the same curve: pops on a
laptop at 44.1/48 kHz, distortion at 88.2/96 kHz, and no sound at all at 176.4
and 192 kHz where the audio thread would need more than real time.

It was not only wasteful. Rendering a 440 Hz tone at zero transpose through the
real processor, the output differed from the input by up to 0.5 full scale with
every bus wired. It is now sample-for-sample identical.

The fix. Keep a per-bus lane count. Wire a pitch bus into the worklet only
while that count is non-zero, wired before the first lane connects and unwired
once the last one leaves. The unpitched bus stays wired for good, because the
click is scheduled onto it whether or not a lane sits there.

This is the handover the processor's own design already describes. In Chrome it
never happened, because every chain was already warm. A chain whose input goes
empty is still fed silence and kept for CHAIN_LINGER_BLOCKS, which is how its
tail drains, so a lane changing key mid-playback hands over with no gap.

It also makes an assumption both engines already relied on true: each returns
zero pipeline latency when no lane is transposed, on the grounds that the worklet
hands its input straight back. That was false on main, where the pitch stage
was always primed, so the playhead ran ahead of the sound at zero transpose.


2. The graph running at the output device's rate (#578)

With the first fix in, an idle graph is cheap at any rate. An active one is
not: a single transposed lane costs 2.1% of real time at 44.1 kHz and 28.2% at
192 kHz.

None of that resolution is doing anything. Every stem the pipeline produces is
44.1 kHz, fixed at -ar 44100 in app/pipeline/runner.py, so a 192 kHz context
upsamples on decode and then works at four times the rate on interpolated
samples. Decoded buffers pay the same multiple: a five minute six stem track
costs 606 MB of AudioBuffers at 44.1 kHz and 2.6 GB at 192 kHz.

The fix. Read the device rate from a throwaway context and, only when it is
above 48 kHz, build the real one with { sampleRate: 48000 }. A cap rather than
a pin: at 44.1 and 48 kHz the graph is left exactly as it is today, because there
is nothing to gain there and a needless resample to lose.

All four creation sites get it, since a fix in one engine is not a fix: both
playback engines, the mobile UI's shared gesture-unlocked context, and the
decode-only context behind the footer waveform, which was holding four times the
buffer to compute the same number of peak bars.

Both fallbacks matter. A browser that cannot build the probe, and one that
refuses the rate we ask for, both end up with the plain constructor. Playing at
the device's rate is what happens today and is never worse than not playing.


Tests

Twelve new checks in the routing test, written by @pywkt, that walk the graph for
which inputs are wired as lanes arrive, share a key and leave. They needed the
mock fix described above to be able to fail.

tests/js/audio-context.test.mjs is new: the cap threshold on both sides, the
probe being closed again, and both fallback paths.

Verification

In headless Chromium with AudioNode.prototype.connect and disconnect wrapped
around the real engine module: main wires all thirteen inputs at load. This
wires [0] at load, [0, 2] after a lane goes to +2, [-5, 0] after moving it
to -5, and [0] when it returns, which reproduces exactly what @pywkt reported
from his own instrumentation. A lane at +2 still renders 494 Hz from a 440 Hz
source. Selective disconnect(node, output, input) confirmed in Chromium and
Firefox. A context whose device reports 192 kHz is built at 48 kHz and still
produces signal.

Verification, and the one thing still open

@wastedbits confirmed the first commit on his own 24-bit/192 kHz Windows
machine.

I then reproduced #575 locally by forcing the graph rate in devtools, since my
interface only exposes 44.1 kHz to Windows shared mode. On main, no sound. On
this branch, clean. The override sets sampleRate after spreading the options,
so it wins over the 48 kHz cap too, which means the branch was running a genuine
192 kHz graph rather than a capped one.

So the first commit is sufficient on its own. Removing the twelve idle
chains is what makes 192 kHz playable; the rate cap is an optimisation on top,
not the thing that stops the silence. That matches the measurements above: 1.5%
of real time at 192 kHz with only the unpitched bus wired, against more than real
time with all thirteen.

That leaves exactly one open question, and it is now a smaller one. Whether
Windows resamples 48 kHz up to 192 kHz cleanly on a real 192 kHz DAC decides
only whether the cap is worth keeping. It no longer decides whether #575 is
fixed. If it turns out to cost anything audible, the second commit can be
dropped and #575 still closes.

Noted, not fixed here

estimateDecodedBytes hardcodes 44100 and no caller passes a rate, so on a
192 kHz device it was underestimating the full-decode engine's memory by 335% and
choosing that engine when it would allocate 2.6 GB. The cap bounds that to an
8.8% underestimate. Worth fixing properly, no longer dangerous.

CI's js-syntax job globs static/js/*.js only, so static/mobile/app.js is
never syntax checked. Checked by hand here.

Gate

55/55 audio-routing, 10/10 audio-context, all tests/js, 127/127 Playwright,
node --check on every file in static/js and static/mobile. No Python, no
uv.lock, no new strings.

Closes #576
Closes #575
Closes #578

🤖 Generated with Claude Code

…n it

Both engines wired all thirteen semitone buses into the SoundTouch worklet
the moment it loaded. The processor decides whether a semitone is in use by
whether its input has channels, and only takes its bypass path when none do.
The comment there says an unconnected input arrives as an empty array, and the
unit test hands unlisted inputs `[]` by construction, so nothing ever
contradicted it.

Chrome does not do that. A bus that is wired with nothing playing into it
arrives as one channel of silence. Measured, with audio on the unpitched bus
only, the processor was handed `[1,1,1,1,1,1,2,1,1,1,1,1,1]`. So all twelve
pitch inputs read as live, twelve PitchChains were built on the first render
quantum, and each ran WSOLA, the 8th-order anti-alias cascade and the
resampler on silence for the whole track. The bypass never engaged, and a
chain is only dropped after 256 blocks of an *empty* input, which never came.

The WSOLA windows are set in milliseconds, so the wasted work per second of
audio scales with the AudioContext's rate, which follows the system output
device. Offline render, seven stems, no transpose, headless Chromium, as a
share of real time:

    rate       only the unpitched bus wired     all thirteen wired
    44.1 kHz   0.2%                             17.8%
    48   kHz   0.3%                             20.9%
    96   kHz   0.8%                             78.4%
    192  kHz   1.5%                             over real time

That is #576 (continuous pops on a laptop at 44.1/48 kHz, fan pinned for the
whole track) and #575 (distortion at 88.2/96 kHz, no sound at all at 176.4 and
192 kHz on Windows) as one bug, seen from two ends of the same curve.

It was not only wasteful. Rendering a 440 Hz tone at zero transpose through
the real processor, the output differed from the input by up to 0.5 full
scale with every bus wired, and is now sample-for-sample identical.

So: keep a per-bus lane count and wire a pitch bus into the worklet only while
that count is non-zero, wired before the first lane connects and unwired once
the last one leaves. The unpitched bus stays wired for good, because the click
is scheduled onto it whether or not a lane sits there. This is the handover the
processor's own design describes; in Chrome it never happened because every
chain was already warm. A chain whose input goes empty is still fed silence and
kept for CHAIN_LINGER_BLOCKS, which is how its tail drains, so a lane changing
key mid-playback hands over with no gap.

It also makes an assumption both engines already relied on true. Each returns
zero pipeline latency when no lane is transposed, on the grounds that the
worklet hands its input straight back. That was false on main, where the pitch
stage was always primed, so the playhead ran ahead of the sound by the priming
latency at zero transpose.

The routing test gains twelve checks that walk the graph for which inputs are
wired as lanes arrive, share a key and leave. `FakeNode.disconnect` ignored its
arguments and cleared every connection, which would have passed a selective
disconnect and a blanket one alike; it now matches the real overloads and
throws on a connection that is not there.

Verified in headless Chromium with `AudioNode.prototype.connect` and
`disconnect` wrapped around the real engine module: `main` wires all thirteen
inputs at load, this wires `[0]` at load, `[0, 2]` after a lane goes to +2,
`[-5, 0]` after moving it to -5, and `[0]` when it returns. A lane at +2 still
renders 494 Hz from a 440 Hz source.

Closes #576
Closes #575

Co-Authored-By: pywkt <90816178+pywkt@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the device's

An AudioContext built with no options runs at whatever the operating system's
output device is set to, and every node downstream inherits it. StemDeck gains
nothing from that above 48 kHz. Every stem the pipeline produces is 44.1 kHz,
fixed at `-ar 44100` in app/pipeline/runner.py, so a 192 kHz context upsamples
on decode and then does several times the work on interpolated samples carrying
nothing the originals did not.

The pitch chain's WSOLA search is O(overlap * seek) per sequence and both
windows are set in milliseconds, so its cost per second of audio is quadratic in
the context rate. One active chain, offline render through the real processor,
as a share of real time:

    44.1 kHz    2.1%
    48   kHz    2.4%
    96   kHz    7.3%
    192  kHz   28.2%

Decoded buffers scale linearly over the same range. A five minute six stem track
costs 606 MB of AudioBuffers at 44.1 kHz and 2.6 GB at 192 kHz, which is enough
to put a tab under memory pressure before any DSP runs.

So read the device rate from a throwaway context and, only when it is above
48 kHz, build the real one with `{ sampleRate: 48000 }`. A cap rather than a
pin: at 44.1 and 48 kHz the graph is left exactly as it is today, because there
is nothing to gain there and a needless resample to lose. The browser resamples
the context to the device on output, which it was doing to our upsampled audio
anyway.

All four creation sites get it, since a fix in one engine is not a fix: both
playback engines, the mobile UI's shared gesture-unlocked context, and the
decode-only context behind the footer waveform, which was holding four times the
buffer to compute the same number of peak bars.

Both fallbacks matter. A browser that cannot build the probe at all, and one
that refuses the rate we ask for, both end up with the plain constructor:
playing at the device's rate is what happens today and is never worse than not
playing.

One thing this does not change but does bound. `estimateDecodedBytes` hardcodes
44100 and no caller passes a rate, so on a 192 kHz device it was underestimating
the full-decode engine's memory by 335% and choosing that engine when it would
allocate 2.6 GB. With the cap in place the worst case is 48 kHz, an 8.8%
underestimate. Worth fixing properly, but no longer dangerous.

Verified in Chromium: a 44.1 kHz device is left at 44.1 kHz, a context whose
device reports 192 kHz is built at 48 kHz, and the capped context still produces
signal. The unit tests cover the cap threshold, the probe being closed again,
and both fallbacks.

Closes #578

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thcp thcp changed the title Stop the pitch worklet running twelve idle chains at zero transpose Stop the pitch worklet burning the audio thread at zero transpose and at high output rates Sep 6, 2026
@thcp

thcp commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Reproduced #575 locally and confirmed the first commit fixes it.

My audio interface only exposes 44.1 kHz to Windows shared mode, so I forced the
graph rate in devtools instead, before opening a track:

const R = window.AudioContext;
window.AudioContext = class extends R {
  constructor(o) { super({ ...o, sampleRate: 192000 }); }
};

Then a six stem track, nothing transposed, speed at 1x.

On main: no sound. Which is exactly what @wastedbits reported at 24-bit/192 kHz.

On this branch: clean.

Worth being precise about what that tests, because the snippet spreads the
options and then overrides sampleRate, so it wins over the 48 kHz cap in the
second commit as well. The branch was therefore running a genuine 192 kHz graph,
not a capped one.

So the first commit is sufficient on its own. Removing the twelve idle chains is
what makes 192 kHz playable, and the rate cap is an optimisation on top rather
than the thing that stops the silence. That lines up with the measurements in
the description: 1.5% of real time at 192 kHz with only the unpitched bus wired,
against more than real time with all thirteen.

This decouples the one open question here. Whether Windows resamples 48 kHz up
to 192 kHz cleanly on a real 192 kHz DAC only decides whether the cap is worth
keeping. It no longer decides whether #575 is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant