Skip to content

fix(rtmg/web): re-dispatch restored LoRA strengths to the engine on session ready#276

Closed
qianghan wants to merge 3 commits into
daydreamlive:mainfrom
qianghan:fix/lora-strength-not-restored-on-reload
Closed

fix(rtmg/web): re-dispatch restored LoRA strengths to the engine on session ready#276
qianghan wants to merge 3 commits into
daydreamlive:mainfrom
qianghan:fix/lora-strength-not-restored-on-reload

Conversation

@qianghan

Copy link
Copy Markdown

Summary

Fixes the tracker bug "Web app bug: LoRAs not loading upon refresh / session reload."

Repro (from the tracker): set a LoRA strength to 1, save the session, reload it. The fader still shows 1, but the audio doesn't sound like the LoRA is applied — it only "kicks in" once you wiggle the fader (drop to 0.8, raise back to 1).

Root cause

LoRA strength has two representations in the web client:

  • useLoraStore.strengths[id] — the source of truth the fader UI renders.
  • usePerformanceStore.sliderValues["lora_str_<id>"] — the engine-facing value that useParamSync actually ships to the pipeline every 8 ms tick.

useParamSync prefers sliderValues and only falls back to lora.strengths when the lora_str_<id> key is absent:

for (const id of lora.enabled) {
  const k = `lora_str_${id}`;
  if (k in raw) continue;           // sliderValues already has it → never read lora.strengths
  const v = lora.strengths[id];
  if (typeof v === "number") raw[k] = v;
}

The boot-time catalog seed (useLoraStore.setCatalogseedLoraSliderValue) writes a sliderValues entry for the default-on LoRAs. A saved-session resume then restores the persisted strength into lora.strengths only — it never updates sliderValues. So at resume the key is present-but-stale: useParamSync keeps streaming the old seeded value and never reads the restored one. The slider shows 1.0 (reads lora.strengths) while the engine stays pinned to the stale strength.

A fader drag is the first thing that writes sliderValues (via the dispatcher's debounced commit), which is exactly why the LoRA only becomes audible after the user moves the fader. Symptom = "shows 1.0 but inaudible"; true cause = the restored strength is never pushed into the engine-facing sliderValues.

This is a state-restoration / initial-sync gap, not a number-persistence gap — the number persists fine.

Fix

  • Add reconcileEnabledLoraStrengths() (engine/lora/dispatcher.ts): for every enabled LoRA, seed perf.sliderValues["lora_str_<id>"] from lora.strengths[id].
  • Call it once per session-ready edge in useStartSession — the initial connect path and the reconnect onSuccess path — right after the LoRA cap is finalized.

Why this is regression-safe

  • No double-apply on a fresh session: on a non-restored session sliderValues and lora.strengths already agree (both seeded from the same config/defaults), so every write lands the same value. The server only refits on a > 0.02 delta, so re-seeding the matching value triggers no extra refit.
  • Never fires before the engine is connected: it runs only on the ready path, with a live remote.
  • No per-tick spam: runs once per ready edge, not in the param loop.
  • Never mid-drag: ready edges can't overlap a pointer gesture; the dispatcher remains the sole owner of sliderValues during a fader drag.
  • Contract-first preserved: no knob/protocol registry was touched (no knobs.py / protocol.py change), so no gen_wire_types.py regeneration is required and the drift guards are unaffected.

Test plan

New focused unit test tests/unit/loraStrengthReconcile.test.ts:

  • restoring a session with LoRA strength = 1 over a stale seeded sliderValues results in the engine receiving 1.0 on ready (mirrors useParamSync's exact selection logic);
  • a fresh session where the values already agree is a no-op;
  • only enabled LoRAs are reconciled.

Evidence:

# typecheck
$ npm run typecheck      → tsc --noEmit, exit 0

# new + lifecycle tests
$ npx vitest run tests/unit/loraStrengthReconcile.test.ts tests/unit/sessionStore.test.ts
  Test Files  2 passed (2)
       Tests  11 passed (11)

# production build
$ npm run build          → ✓ Compiled successfully, exit 0

Note: tests/unit/float16.test.ts and tests/unit/sliceEpoch.test.ts fail locally with TypeError: Math.f16round is not a function — a pre-existing Node-version limitation in the float16 wire helpers, unrelated to this change (no LoRA/param code involved).

Made with Cursor

seanhanca and others added 3 commits June 17, 2026 10:37
…ing an outage doesn't keep playing stale

When the WS drops and the user switches to a different audio source before
the auto-reconnect completes, the recovered session kept playing the
previously-loaded track even though the UI showed the new selection.

Root cause: the reconnect path (useStartSession) rebinds the fixture
snapshotted at session start. The mid-outage track change is dropped
because useFixtureSwap.run() bails while status !== "ready", and nothing
re-applies it once the session recovers — so the recovered backend +
AudioPlayer stay bound to the stale track while the perf store's fixture
shows the new pick.

Fix: track the live session's bound fixture in useSessionStore
(boundFixture), set on initial connect and reconnect. useFixtureSwap now
reconciles on the "reconnecting" -> "ready" edge: if the selected fixture
diverged from the bound one, it re-runs the swap exactly once. No effect
on a fresh Play or a clean reconnect where the selection never changed.

Adds needsFixtureReconcile() + a focused unit test pinning the decision
matrix, and asserts boundFixture clears on session reset.

Co-authored-by: Cursor <cursoragent@cursor.com>
… clip

A fast knob sweep landed as a raw discontinuity on consecutive
generations (the runner reads the knob target once per tick and the
backend turns it straight into per-inference-step values), driving the
audio into distortion/clipping. There was no hysteresis or rate-limiting
on control changes.

Add an engine-side per-knob slew limiter at the once-per-tick knob read
(ACEStepBackend.read_knobs), so a large jump ramps over a few ticks
instead of snapping. Applied behind the GeneratorBackend seam, it
protects every transport (web, VST, MCP, headless) with one
implementation rather than per-UI tweens.

Contract-first: the rate ceiling is registry metadata
(KnobSpec.slew_max_per_s, projected into the /api/knobs catalog and
bumped to KNOB_SCHEMA_VERSION 2). Default is range-relative
(DEFAULT_SLEW_FRACTION_PER_S = 3.0, a full-range sweep ramps over
~0.33s) and applies only to continuous float knobs; int/enum/bool knobs
(seed, step count, modes, toggles) and non-registry keys (curve specs,
the playback clock) pass through verbatim, so the rebuild signature and
discrete controls are unaffected. A slew limiter caps only the rate of
change, so normal/slow adjustments incur zero added latency.

hint_strength and x0_target now read from the slewed knob dict instead
of the backing KnobState directly, so the whole translation path honors
the limit.

Regenerated packages/demon-client/types/wireContract.gen.ts; added
tests/unit/test_knob_slew.py (bounded per-tick deltas on a jump,
discrete/unknown passthrough, no-lag slow changes, dt cap, catalog
projection).

Co-authored-by: Cursor <cursoragent@cursor.com>
…ession ready

A saved-session resume restores the persisted LoRA strength into
useLoraStore.strengths (which the fader UI reads, so it shows e.g. 1.0)
but never into perf.sliderValues — the value useParamSync actually
ships to the engine each tick. useParamSync prefers sliderValues and
only falls back to lora.strengths when the lora_str_<id> key is ABSENT,
so the boot-time catalog seed's stale entry pins the engine to the old
strength: the LoRA shows its restored strength but stays inaudible
until a fader drag finally writes sliderValues through the dispatcher.

Add reconcileEnabledLoraStrengths(), which seeds the engine-facing
slider value of every enabled LoRA from the store strength, and call it
once per session-ready edge (initial connect + reconnect) in
useStartSession. Idempotent on a fresh session (values already agree →
no extra server refit); gated on the ready path so it never fires
before the engine is connected and never spams per-tick dispatches.

No contract registry touched, so no wire-type regeneration is needed.

Co-authored-by: Cursor <cursoragent@cursor.com>
@seanhanca

Copy link
Copy Markdown

Superseded by #280, re-opened from a branch on this repo (daydreamlive/DEMON) instead of the qianghan fork. Closing this fork-based PR in favor of #280.

@seanhanca seanhanca closed this Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants