diff --git a/apps/cadecon/PLAN.md b/apps/cadecon/PLAN.md index 32adcb0d..0ec37be1 100644 --- a/apps/cadecon/PLAN.md +++ b/apps/cadecon/PLAN.md @@ -186,7 +186,7 @@ - 20×20 log-spaced grid search over (tau_r, tau_d) - Closed-form beta per grid point - Golden-section refinement (not Nelder-Mead) - - `BiexpResult { tau_rise, tau_decay, beta, residual }` + - `BiexpResult { tau_rise, tau_decay, beta, residual, tau_rise_fast, tau_decay_fast, beta_fast, fit_mode }` (two-component slow+fast model; `fit_mode` is a `FitMode` enum — `TwoComponent` / `SlowOnly` / `Degenerate` / `Empty` — reporting the outcome) - tau_d > tau_r enforced, validity checks - 6 tests (recovers known taus, tau_d > tau_r enforced, refinement improves fit, positive beta, empty kernel, various parameter ranges) @@ -199,6 +199,7 @@ - `indeca_compute_upsample_factor(fs, target_fs) -> usize` - Added `serde` + `serde-wasm-bindgen` optional deps to Cargo.toml - `InDecaResult` and `BiexpResult` derive `Serialize` when `jsbindings` feature is active + - **Non-finite input guard:** the WASM entry points reject input traces containing `NaN`/`Inf` (throw a JS error via `crate::first_nonfinite`) instead of propagating garbage results ### 2.7 TypeScript: CaDecon worker — ✅ DONE @@ -221,8 +222,9 @@ 2. Per-subset kernel estimation (parallel kernel-jobs via pool) 3. Merge: median tauRise/tauDecay across subsets 4. Record in convergenceHistory, update currentTauRise/currentTauDecay - 5. Convergence check: `max(|Δτ_r|/τ_r, |Δτ_d|/τ_d) < convergenceTol` - 6. Finalization pass: re-run trace inference on ALL cells with converged kernel + 5. Convergence check (kernel **shape space**, not tau deltas — (τ_r, τ_d) is a degenerate coordinate): an iteration is "stable" when the relative change of BOTH the kernel peak time (tPeak) AND FWHM is `< convergenceTol`; convergence is declared after `convergencePatience` consecutive stable iterations, gated by `convergenceMinIters`. Defaults come from `@calab/core` `CONVERGENCE_RANGES`. + 6. Final kernel = median of the last `finalSelectionWindow` iterates' shapes + 7. Finalization pass: re-run trace inference on ALL cells with the converged kernel - **pauseRun():** sets `runState('paused')`, blocks loop via Promise resolver - **resumeRun():** resolves pause Promise, sets `runState('running')` - **stopRun():** `pool.cancelAll()`, resolves pause, stores intermediate results @@ -295,21 +297,24 @@ ### 3.1 Kernel convergence plot — ✅ DONE - [x] `src/components/charts/KernelConvergence.tsx` — **rewrote** from canvas to SolidUplot - - Left Y-axis: tau_rise + tau_decay (ms) as line series with dot markers - - Right Y-axis (secondary): residual on `'res'` scale, dashed gray line + - Now the **"Kernel" tab** of the consolidated `ConvergencePanel.tsx` (tabs: Asymptote / Kernel / Distributions), not a standalone panel + - Left Y-axis: tau_rise + tau_decay (ms) as line series with dot markers, plus the shape-space metrics (peak time, FWHM) that drive convergence + - Residual series **removed** (convergence is judged in shape space, not on residual) - Per-subset scatter: custom `draw` hook plugin draws faint circles behind median lines - Convergence marker: `convergence-marker-plugin.ts` draws vertical dashed green line at `convergedAtIteration()` - Empty state: ` 0}>` gate with placeholder text - Wheel zoom + cursor sync key `'cadecon-convergence'` + - Sibling tab `AsymptoteTrends.tsx` — small-multiples of the four convergence signals (asymptote dashboard) ### 3.2 Kernel shape display — ✅ DONE - [x] **Deleted** `src/components/charts/DebugKernelChart.tsx` - [x] **Created** `src/components/kernel/KernelDisplay.tsx` — uPlot chart with: - - Per-subset h_free as faint colored lines (D3 category10 with 0.4 opacity) - - Merged bi-exponential fit as bold dashed purple line + - Per-subset h_free as faint colored lines (shared colorblind-safe Okabe-Ito palette from `@calab/ui/chart`, low opacity) + - Merged bi-exponential fit as a bold dashed line (merged-fit color from the shared palette) - X-axis: time in ms, Y-axis: amplitude - DOM overlay stats: tau_r, tau_d, beta values + - Degeneracy warning: shows `⚠ {bad}/{total} fits degenerate` when subset fits report `FitMode::Degenerate`/`Empty` - Reads `viewedIteration()` from viz-store (null = latest) - Handles dynamic subset count via recreating series config - Cursor sync key `'cadecon-kernel'` @@ -390,13 +395,15 @@ ### 3.9 Chart infrastructure — ✅ DONE - [x] Added `uplot` + `@dschz/solid-uplot` to `apps/cadecon/package.json` -- [x] `src/lib/chart/series-config.ts` — 11 series factories + `withOpacity` helper + D3 category10 palette +- [x] `src/lib/chart/series-config.ts` — 11 series factories, now sourcing colors from the shared colorblind-safe Okabe-Ito palette in `@calab/ui/chart` (was a local D3 category10 palette) - [x] `src/lib/chart/chart-theme.css` — uPlot theme overrides (copied from CaTune) - [x] `src/lib/chart/wheel-zoom-plugin.ts` — scroll zoom + drag pan (copied from CaTune) - [x] `src/lib/chart/theme-colors.ts` — CSS custom property reader (adapted for CaDecon accent) - [x] `src/lib/chart/transient-zone-plugin.ts` — pad zone shading (copied from CaTune) - [x] `src/lib/chart/convergence-marker-plugin.ts` — convergence vertical dashed line +> **Later centralized (PRs #158–160):** the palette (Okabe-Ito), viridis colormap, tick math (`niceTicks`), and uPlot axis/cursor/range helpers (`chartAxis`, `labeledAxis`, `syncCursor`, `safeRange`) now live in the shared `@calab/ui/chart` module rather than as CaDecon-local copies. `series-config.ts` and the CaDecon charts consume them from there. + ### 3.10 CSS — ✅ DONE - [x] `src/styles/layout.css` — `.viz-grid` 3-row flex layout with responsive stacking @@ -537,6 +544,7 @@ CaDecon uses banded AR(2) convolution exclusively (no FFT). The existing `Banded - `convolve_forward(s, c)`: given spikes s, produce calcium c - `convolve_adjoint(r, g)`: transpose operation for FISTA gradient - AR(2) coefficients computed from bi-exponential (tau_r, tau_d) at the upsampled sampling rate +- **One-sample source delay** (`output[t] += source[t-1] * inv_peak`): the AR(2) forward model is aligned so its impulse response matches the reference `build_kernel` double-exponential (which is 0 at t=0). The same delay is mirrored in `apps/cadecon/src/lib/reconvolve.ts` so the on-demand reconvolution overlay stays aligned with the solver's forward model. ### Threshold Search Strategy (Designed) @@ -724,7 +732,7 @@ crates/solver/src/ ### Raster rendering -- `RasterOverview.tsx` uses a viridis LUT (11-stop linear interpolation, 256 entries) with 1st–99th percentile scaling. +- `RasterOverview.tsx` uses the shared `VIRIDIS_LUT` (256 entries) from `@calab/ui/chart` with 1st–99th percentile scaling, and shared `niceTicks` for the axes. The intensity **colorbar was intentionally removed** — activity is assumed to span 0→full and the absolute values are not meaningful. - Canvas renders at `devicePixelRatio` for HiDPI. Height is `min(max(200, N*3), 500)` pixels. - Click detection uses pixel-to-data coordinate mapping against `subsetRectangles`. @@ -778,7 +786,7 @@ The iteration loop runs on the main thread and dispatches jobs to the worker poo 1. **Per-trace inference**: dispatches `TraceJob` for each cell in each subset rectangle 2. **Kernel estimation**: dispatches `KernelJob` per subset with concatenated traces/spikes 3. **Merge**: median of subset tau estimates (no informativeness weighting yet) -4. **Convergence**: `max(|Δτ_r|/τ_r, |Δτ_d|/τ_d) < convergenceTol` +4. **Convergence**: shape-space test — both kernel peak time (tPeak) and FWHM must change by `< convergenceTol` for `convergencePatience` consecutive iterations (gated by `convergenceMinIters`); final kernel = median of the last `finalSelectionWindow` iterates' shapes. Defaults in `@calab/core` `CONVERGENCE_RANGES`. 5. **Finalization**: re-runs all cells with converged kernel Pause/resume uses a Promise-based mechanism — the loop `await`s a resolver that only fires on resume. @@ -817,9 +825,9 @@ The canvas in `KernelConvergence.tsx` must be `position: absolute` inside a `pos ### Chart infrastructure -All chart files live in `src/lib/chart/`. Four files were copied from CaTune (`chart-theme.css`, `wheel-zoom-plugin.ts`, `theme-colors.ts`, `transient-zone-plugin.ts`) with minor adjustments (accent color default in theme-colors). Two new files were created: `series-config.ts` (CaDecon-specific series factories with D3 category10 palette) and `convergence-marker-plugin.ts`. +CaDecon-local chart files live in `src/lib/chart/`: `series-config.ts` (CaDecon-specific series factories) and `convergence-marker-plugin.ts`. The palette, viridis colormap, tick math, uPlot theme, axis/cursor/range helpers, and zoom/transient-zone plugins are now shared via `@calab/ui/chart` (PRs #158–160) rather than kept as CaTune copies; `series-config.ts` sources its colors from the shared Okabe-Ito palette there. -`TracePanel.tsx` was also copied from CaTune into `src/components/traces/` with import path adjustments. +The reusable `TracePanel` now lives in `@calab/ui/chart`; CaDecon's `src/components/traces/` holds the app-specific `CellSelector.tsx`, `IterationScrubber.tsx`, and `TraceInspector.tsx`. ### uPlot integration pattern diff --git a/apps/catune/README.md b/apps/catune/README.md index 05e6825e..80e213d8 100644 --- a/apps/catune/README.md +++ b/apps/catune/README.md @@ -83,11 +83,11 @@ Components are organized by feature area under `src/components/`: `cards/`, `com ### Internal Packages -| Package | Role in CaTune | -| ------------------ | -------------------------------------------------------------------------------------- | -| `@calab/core` | Shared types, WASM adapter (`initWasm`, `Solver`), metrics, FFT, parameter ranges | -| `@calab/compute` | Worker pool, warm-start cache, kernel math, downsampling, demo presets, synthetic data | -| `@calab/io` | .npy/.npz parsing, trace validation, cell ranking, JSON export | -| `@calab/community` | Supabase client, CRUD operations, submission logic, field options | -| `@calab/tutorials` | Tutorial types, progress persistence, tutorial engine | -| `@calab/ui` | DashboardShell, DashboardPanel, VizLayout, CompactHeader, CardGrid | +| Package | Role in CaTune | +| ------------------ | -------------------------------------------------------------------------------------------------------- | +| `@calab/core` | Shared types, WASM adapter (`initWasm`, `Solver`), metrics, FFT, parameter ranges | +| `@calab/compute` | Worker pool, warm-start cache, kernel math, downsampling, demo presets, synthetic data | +| `@calab/io` | .npy/.npz parsing, trace validation, cell ranking, JSON export | +| `@calab/community` | Supabase client, CRUD operations, submission logic, field options | +| `@calab/tutorials` | Tutorial types, progress persistence, tutorial engine | +| `@calab/ui` | DashboardShell, DashboardPanel, VizLayout, CompactHeader, CardGrid, chart primitives (`@calab/ui/chart`) | diff --git a/crates/solver/README.md b/crates/solver/README.md index a362fb95..41a50147 100644 --- a/crates/solver/README.md +++ b/crates/solver/README.md @@ -4,7 +4,14 @@ Rust FISTA deconvolution solver with dual WASM/PyO3 targets. ## Overview -This crate implements the FISTA (Fast Iterative Shrinkage-Thresholding Algorithm) solver used by CaTune for calcium trace deconvolution. It is compiled to WebAssembly via `wasm-pack` and runs in Web Workers in the browser. The compiled output in `pkg/` is committed to the repository so that CI and development do not require a Rust toolchain. +This crate implements calcium trace deconvolution: a FISTA (Fast Iterative Shrinkage-Thresholding Algorithm) core plus the higher-level InDeCa pipeline (rolling-baseline estimation, free-form kernel estimation, bi-exponential kernel fitting, peak-seeded bootstrap, and up/down-sampling) used by CaTune and CaDecon. + +It builds two ways, gated by Cargo features: + +- **`jsbindings`** (default) — compiled to WebAssembly via `wasm-pack` and run in Web Workers in the browser. The compiled output in `pkg/` is committed to the repository so that CI and development do not require a Rust toolchain. +- **`pybindings`** — compiled as a native PyO3 extension module for the `calab` Python package (see `python/`). + +`cargo test` uses the default (`jsbindings`); the PyO3 surface is checked separately with `--no-default-features --features pybindings`. ## Algorithm @@ -14,14 +21,14 @@ The solver minimizes the following objective with a non-negativity constraint: minimize (1/2)||y - K·s - b||² + λ·G_dc·||s||₁ subject to s ≥ 0 ``` -| Symbol | Meaning | -| ------ | ------------------------------------------------------------------------------------------ | -| `y` | Input fluorescence trace | -| `K` | Convolution matrix from double-exponential kernel | -| `s` | Deconvolved activity (output) | -| `b` | Scalar baseline, estimated jointly as `mean(y - K·s)` | -| `λ` | Sparsity penalty (user-adjustable) | -| `G_dc` | Kernel DC gain `Σh`, scales λ so the sparsity slider is effective across all kernel shapes | +| Symbol | Meaning | +| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `y` | Input fluorescence trace | +| `K` | Convolution matrix from double-exponential kernel | +| `s` | Deconvolved activity (output) | +| `b` | Scalar baseline. Estimated per-iteration as `mean(y - K·s)`, then EMA-smoothed for a stable display value (`BASELINE_EMA_WEIGHT`). The InDeCa driver instead removes a rolling-quantile floor up front (`baseline::DEFAULT_BASELINE_QUANTILE`) so the baseline is ~0 and this term can be skipped. | +| `λ` | Sparsity penalty (user-adjustable) | +| `G_dc` | Kernel DC gain `Σh`, scales λ so the sparsity slider is effective across all kernel shapes | **Kernel:** `h(t) = exp(-t/τ_decay) - exp(-t/τ_rise)`, normalized to peak = 1.0. Length extends until the decay envelope drops below 1e-6 of peak. @@ -29,42 +36,92 @@ minimize (1/2)||y - K·s - b||² + λ·G_dc·||s||₁ subject to s ≥ 0 **Adaptive restart:** O'Donoghue & Candes (2015) gradient-mapping criterion — resets momentum to `t = 1` when the proximal step undoes the momentum direction. -**Convergence:** Primal residual criterion `||x_{k+1} - x_k|| / ||x_k|| < 1e-6` after iteration 5. This avoids an expensive forward convolution + objective evaluation per iteration. +**Convergence (inner FISTA loop):** Primal residual criterion `||x_{k+1} - x_k|| / ||x_k|| < tol` after iteration 5, where `tol` defaults to `1e-4` and is configurable. This avoids an expensive forward convolution + objective evaluation per iteration. (The _outer_ InDeCa iteration — alternating spike solve and kernel re-estimation — instead converges in kernel shape space: it stops when the kernel's peak time and FWHM both reach an asymptote. Those controls live in `@calab/core` `CONVERGENCE_RANGES`.) + +**Forward model:** two interchangeable convolution engines selected via `set_conv_mode` — `ConvMode::Fft` (O(n log n) DFT) and `ConvMode::BandedAR2` (O(n) banded AR(2) recursion). The banded engine applies a one-sample source delay so its output stays aligned with the double-exponential `build_kernel` reference (mirrored in `apps/cadecon/src/lib/reconvolve.ts`). ## Modules -| Module | Description | -| ----------- | ------------------------------------------------------------------------------------------------------------------------ | -| `lib.rs` | `Solver` struct — public wasm-bindgen API, parameter management, state serialization, bandpass filter methods | -| `kernel.rs` | `build_kernel` (double-exponential), `compute_lipschitz` (spectral bound via DFT) | -| `fista.rs` | `step_batch` — FISTA iteration loop with FFT convolutions, adaptive restart, convergence check | -| `fft.rs` | `FftConvolver` — self-contained FFT convolution engine with pre-computed kernel spectrum, forward and adjoint operations | -| `filter.rs` | `BandpassFilter` — FFT-based bandpass filter derived from kernel time constants, cosine-tapered transitions | +### Core FISTA + +| Module | Description | +| -------------- | ------------------------------------------------------------------------------------------------------------------------ | +| `lib.rs` | `Solver` struct — parameter management, state serialization, bandpass/baseline methods, `first_nonfinite` FFI guard | +| `kernel.rs` | `build_kernel` (double-exponential), `compute_lipschitz` (spectral bound via DFT) | +| `fista.rs` | `step_batch` — FISTA iteration loop with adaptive restart and convergence check | +| `fft.rs` | `FftConvolver` — self-contained FFT convolution engine with pre-computed kernel spectrum, forward and adjoint operations | +| `banded.rs` | `BandedAR2` — O(n) banded AR(2) forward/adjoint convolution engine (one-sample source-delay aligned) | +| `filter.rs` | `BandpassFilter` — FFT-based bandpass filter derived from kernel time constants, cosine-tapered transitions | +| `baseline.rs` | Rolling-quantile baseline estimation/subtraction; `DEFAULT_BASELINE_QUANTILE` | +| `threshold.rs` | Threshold/proximal helpers | + +### InDeCa pipeline + +| Module | Description | +| --------------- | ------------------------------------------------------------------------------------------------------ | +| `indeca.rs` | InDeCa driver — alternating single-trace spike solve and kernel re-estimation | +| `kernel_est.rs` | `estimate_free_kernel` — free-form kernel estimation from traces + spike trains (TV-L1 smoothing) | +| `biexp_fit.rs` | `fit_biexponential` — two-component bi-exponential fit to a free-form kernel; `BiexpResult`, `FitMode` | +| `peak_seed.rs` | Peak-seeded bootstrap — `SeedConfig`, `find_seed_spikes`, `seed_trace`, `seed_kernel_estimate` | +| `upsample.rs` | Up/down-sampling and `compute_upsample_factor` | +| `simulate.rs` | Synthetic trace simulation (Markov/Poisson spiking, kernel, noise, photobleaching, saturation) | + +### FFI bindings + +| Module | Feature | Description | +| ---------------- | ------------ | --------------------------------------------------------------------- | +| `js_indeca.rs` | `jsbindings` | wasm-bindgen free functions for the InDeCa/seed/biexp pipeline | +| `js_simulate.rs` | `jsbindings` | wasm-bindgen simulation entry points | +| `py_api.rs` | `pybindings` | PyO3 `Solver` class + module functions for the `calab` Python package | ## Public API -Methods exposed to JavaScript via `wasm-bindgen`: - -| Method | Description | -| -------------------------------------------------- | ------------------------------------------------------------------------------- | -| `new()` | Create solver with default parameters (τ_rise=0.02, τ_decay=0.4, λ=0.01, fs=30) | -| `set_params(tau_rise, tau_decay, lambda, fs)` | Update parameters and rebuild kernel | -| `set_trace(trace)` | Load a trace, grow buffers if needed, reset iteration state | -| `step_batch(n_steps)` | Run N FISTA iterations, return true if converged | -| `get_solution()` | Get deconvolved activity (owned copy) | -| `get_reconvolution()` | Get K·s (lazy-computed, owned copy) | -| `get_reconvolution_with_baseline()` | Get K·s + b (owned copy) | -| `get_baseline()` | Get estimated scalar baseline | -| `get_trace()` | Get current trace (may be filtered) | -| `converged()` | Check convergence flag | -| `iteration_count()` | Get iteration count | -| `reset_momentum()` | Reset FISTA momentum for warm-start after kernel change | -| `export_state()` / `load_state(state)` | Serialize/restore solver state for warm-start cache | -| `set_filter_enabled(enabled)` / `filter_enabled()` | Toggle bandpass filter | -| `apply_filter()` | Apply bandpass filter to loaded trace | -| `get_power_spectrum()` | Get \|FFT\|² of current trace | -| `get_spectrum_frequencies()` | Get frequency axis in Hz | -| `get_filter_cutoffs()` | Get [f_hp, f_lp] cutoff frequencies | +### `Solver` methods (wasm-bindgen) + +| Method | Description | +| --------------------------------------------------------- | ------------------------------------------------------------------------------- | +| `new()` | Create solver with default parameters (τ_rise=0.02, τ_decay=0.4, λ=0.01, fs=30) | +| `set_params(tau_rise, tau_decay, lambda, fs)` | Update parameters and rebuild kernel | +| `set_trace(trace)` | Load a trace, grow buffers if needed, reset iteration state | +| `set_conv_mode(mode)` | Select the forward-model engine (`Fft` or `BandedAR2`) | +| `set_constraint(c)` | Select the proximal constraint (`NonNegative` L1 or `Box01`) | +| `get_kernel()` | Get the current double-exponential kernel | +| `set_hp_filter_enabled(on)` / `set_lp_filter_enabled(on)` | Toggle the high-/low-pass halves of the bandpass filter individually | +| `step_batch(n_steps)` | Run N FISTA iterations, return true if converged | +| `get_solution()` | Get deconvolved activity (owned copy) | +| `get_reconvolution()` | Get K·s (lazy-computed, owned copy) | +| `get_reconvolution_with_baseline()` | Get K·s + b (owned copy) | +| `get_baseline()` | Get estimated scalar baseline | +| `get_trace()` | Get current trace (may be filtered) | +| `converged()` | Check convergence flag | +| `iteration_count()` | Get iteration count | +| `reset_momentum()` | Reset FISTA momentum for warm-start after kernel change | +| `export_state()` / `load_state(state)` | Serialize/restore solver state for warm-start cache | +| `set_filter_enabled(enabled)` / `filter_enabled()` | Toggle bandpass filter | +| `apply_filter()` | Apply bandpass filter to loaded trace | +| `get_power_spectrum()` | Get \|FFT\|² of current trace | +| `get_spectrum_frequencies()` | Get frequency axis in Hz | +| `get_filter_cutoffs()` | Get [f_hp, f_lp] cutoff frequencies | + +### InDeCa pipeline functions (wasm-bindgen) + +Free functions in `js_indeca.rs`, exposed alongside `Solver`: + +| Function | Description | +| ----------------------------------------------- | ------------------------------------------------------------------------ | +| `indeca_solve_trace(...)` | Solve a single trace (spikes + alpha + baseline + PVE + convergence) | +| `indeca_estimate_kernel(...)` | Estimate a free-form kernel from traces and their spike trains | +| `indeca_fit_biexponential(...)` | Fit a two-component bi-exponential to a free-form kernel → `BiexpResult` | +| `indeca_compute_upsample_factor(fs, target_fs)` | Integer up-sampling factor | +| `seed_trace(trace, fs)` | Peak-seeded bootstrap for a single trace | + +**Non-finite input guard:** the FFI entry points (both wasm-bindgen and PyO3) reject input traces containing `NaN`/`±Inf` — WASM throws a JS error, PyO3 raises `ValueError` — rather than letting a non-finite value propagate into garbage results. + +**Bi-exponential fit outcome:** `BiexpResult` carries a `FitMode` — `TwoComponent`, `SlowOnly`, `Degenerate` (no positive slow amplitude), or `Empty` (no fit) — so callers can detect an untrustworthy fit instead of inferring it. Over PyO3, `fit_biexponential` returns an 8-tuple whose trailing element is the `fit_mode` string. + +### Python API (PyO3) + +Built with the `pybindings` feature and consumed by the `calab` package. Exposes a `Solver` `#[pyclass]` plus module functions (`deconvolve_single`, `deconvolve_batch`, `build_kernel`, `compute_lipschitz`, `solve_trace`, `estimate_kernel`, `fit_biexponential`, `seed_trace`, `seed_kernel_estimate`, `compute_upsample_factor`). See `python/docs/` for the Python-facing reference. ## Build @@ -90,9 +147,11 @@ npm run build:wasm ## Dependencies -| Crate | Purpose | -| -------------------------- | ------------------------------------------ | -| `wasm-bindgen` | JavaScript interop | -| `console_error_panic_hook` | Readable panic messages in browser console | -| `realfft` | Real-valued FFT (wraps rustfft) | -| `rustfft` | FFT computation | +| Crate | Purpose | +| -------------------------- | ----------------------------------------------------- | +| `wasm-bindgen` | JavaScript interop | +| `console_error_panic_hook` | Readable panic messages in browser console | +| `realfft` | Real-valued FFT (wraps rustfft) | +| `rustfft` | FFT computation | +| `pyo3` / `numpy` | PyO3 extension + NumPy interop (`pybindings` feature) | +| `serde` / `serde_json` | Result serialization for the FFI layers | diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 50f013c2..f41d7e0d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -6,7 +6,7 @@ CaLab is a monorepo of calcium imaging analysis tools built with SolidJS, TypeSc ## Monorepo Structure -CaLab uses npm workspaces with seven packages and two applications: +CaLab uses npm workspaces with six built packages and four applications (plus an `apps/_template`). The tree below details `catune` and `carank`; the other apps (`cadecon`, `admin`) follow the same `src/{App.tsx,components,lib,workers,styles}` shape. `apps/cala` and `packages/cala-*` are reserved scaffolding for the in-progress streaming-cala port and are not yet built. ``` . @@ -51,6 +51,9 @@ CaLab uses npm workspaces with seven packages and two applications: │ ├── vite.config.ts │ ├── tsconfig.json │ └── package.json +│ ├── cadecon/ # SolidJS SPA — automated InDeCa deconvolution (kernel + params from data) +│ └── admin/ # SolidJS SPA — community-submission admin +│ # apps/cala/ — reserved scaffolding for the streaming-cala port (not yet built) ├── packages/ │ ├── core/ # @calab/core — shared types, pure math, WASM adapter │ │ └── src/ @@ -59,7 +62,7 @@ CaLab uses npm workspaces with seven packages and two applications: │ │ ├── solver-types.ts # Worker protocol types │ │ ├── types.ts # NpyResult, ValidationResult, etc. │ │ ├── ar2.ts # AR(2) coefficient derivation -│ │ ├── param-config.ts # Parameter ranges +│ │ ├── param-config.ts # Parameter ranges + CONVERGENCE_RANGES (shape-space convergence defaults) │ │ ├── format-utils.ts # Number formatting │ │ ├── metrics/ # snr.ts, solver-metrics.ts │ │ ├── spectrum/ # fft.ts (periodogram computation) @@ -96,16 +99,22 @@ CaLab uses npm workspaces with seven packages and two applications: │ │ ├── index.ts │ │ ├── types.ts # Tutorial, TutorialStep, TutorialProgress │ │ └── progress.ts # localStorage persistence -│ └── ui/ # @calab/ui — shared layout components -│ └── src/ -│ ├── index.ts # Barrel: DashboardShell, DashboardPanel, VizLayout -│ ├── DashboardShell.tsx # 3-section grid (header/main/sidebar) -│ ├── DashboardPanel.tsx # Variant-based panel wrapper -│ ├── VizLayout.tsx # Scroll/dashboard mode switcher -│ └── styles/ -│ └── layout.css # Layout CSS rules +│ ├── ui/ # @calab/ui — shared layout + chart primitives +│ │ └── src/ +│ │ ├── index.ts # Barrel: layout components + chart re-exports +│ │ ├── DashboardShell.tsx # 3-section grid (header/main/sidebar) +│ │ ├── DashboardPanel.tsx # Variant-based panel wrapper +│ │ ├── VizLayout.tsx # Scroll/dashboard mode switcher +│ │ ├── chart/ # Shared uPlot primitives (subpath @calab/ui/chart) +│ │ │ ├── series-utils.ts # Okabe-Ito colorblind palette (TRACE_COLORS, …), subsetColor +│ │ │ ├── colormap.ts # VIRIDIS_LUT, viridisRGB/viridisCss +│ │ │ ├── chart-math.ts # niceTicks +│ │ │ └── axis-helpers.ts # chartAxis, labeledAxis, syncCursor, safeRange +│ │ └── styles/ +│ │ └── layout.css # Layout CSS rules +│ # cala-core/, cala-runtime/ — reserved for the streaming-cala port (not yet built) ├── crates/ -│ └── solver/ # Rust FISTA solver crate (WASM + PyO3) +│ └── solver/ # Rust FISTA + InDeCa solver crate (WASM + PyO3) │ └── pkg/ # wasm-pack output (committed) ├── supabase/ # Supabase config ├── python/ # Python utilities @@ -126,23 +135,30 @@ CaLab uses npm workspaces with seven packages and two applications: @calab/io ← @calab/core @calab/community ← @calab/core @calab/tutorials ← leaf (no local deps) -@calab/ui ← leaf (solid-js only) +@calab/ui ← leaf (solid-js + uplot only) apps/catune ← all packages apps/carank ← @calab/core, @calab/io, @calab/ui +apps/cadecon ← @calab/core, @calab/io, @calab/ui, @calab/compute, @calab/community, @calab/tutorials +apps/admin ← @calab/community, @calab/ui ``` +> The exact per-app dependency set is authoritative in each app's `package.json`; +> the lines above summarize the intended layering. + ## Package Responsibilities -| Package | Responsibility | Key deps | -| ------------------ | ---------------------------------------------------------- | --------------------------------------- | -| `@calab/core` | Shared types, pure utilities, domain math, WASM adapter | `valibot` | -| `@calab/compute` | Generic worker pool, warm-start caching | `@calab/core` | -| `@calab/io` | File parsers (.npy/.npz), data validation, JSON export | `@calab/core`, `fflate`, `valibot` | -| `@calab/community` | Supabase DAL, submission logic, field options | `@calab/core`, `@supabase/supabase-js` | -| `@calab/tutorials` | Tutorial type definitions, progress persistence | none | -| `@calab/ui` | Shared layout: DashboardShell, DashboardPanel, VizLayout | `solid-js` | -| `apps/catune` | SolidJS app — UI components, reactive stores, worker entry | all packages | -| `apps/carank` | SolidJS app — CNMF trace quality ranking | `@calab/core`, `@calab/io`, `@calab/ui` | +| Package | Responsibility | Key deps | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | +| `@calab/core` | Shared types, pure utilities, domain math, WASM adapter | `valibot` | +| `@calab/compute` | Generic worker pool, warm-start caching | `@calab/core` | +| `@calab/io` | File parsers (.npy/.npz), data validation, JSON export | `@calab/core`, `fflate`, `valibot` | +| `@calab/community` | Supabase DAL, submission logic, field options | `@calab/core`, `@supabase/supabase-js` | +| `@calab/tutorials` | Tutorial type definitions, progress persistence | none | +| `@calab/ui` | Shared layout (DashboardShell, DashboardPanel, VizLayout) + chart primitives (`@calab/ui/chart`: palette, colormap, axis/cursor helpers) | `solid-js`, `uplot` | +| `apps/catune` | SolidJS app — deconvolution parameter tuning | all packages | +| `apps/carank` | SolidJS app — CNMF trace quality ranking | `@calab/core`, `@calab/io`, `@calab/ui` | +| `apps/cadecon` | SolidJS app — automated InDeCa deconvolution | core, io, ui, compute, community, tutorials | +| `apps/admin` | SolidJS app — community-submission admin | `@calab/community`, `@calab/ui` | Packages export pure logic. The app wires packages to SolidJS signals. diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 23fc655b..e88ece4d 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -5,13 +5,43 @@ Versions correspond to git tags (`v*`) and apply to the entire monorepo. ## [Unreleased] +> Note: entries between PR #59 and #156 were not recorded here as they merged; +> the list below resumes changelog coverage from the CaDecon pre-publication +> review series (PRs #156–163). + +### Added + +- **CaDecon** bi-exponential fit outcome surfaced as `FitMode` + (`TwoComponent` / `SlowOnly` / `Degenerate` / `Empty`) on the kernel result; + Python `fit_biexponential` now returns an 8-tuple (trailing `fit_mode` string) + and `BiexpFitResult` gained a `fit_mode` field; KernelDisplay warns when + subset fits are degenerate (PR #162) +- Shared uPlot chart primitives in `@calab/ui/chart` — colorblind-safe + Okabe-Ito palette (`TRACE_COLORS`, `GROUND_TRUTH_COLORS`, `KERNEL_FIT_COLORS`, + `METRIC_COLORS`, `subsetColor`), viridis colormap (`VIRIDIS_LUT`, + `viridisRGB`/`viridisCss`), tick math (`niceTicks`), and axis/cursor/range + helpers (`chartAxis`, `labeledAxis`, `syncCursor`, `safeRange`) (PRs #158, #159, #160) +- Comprehensive README files across all packages and apps (PR #58) + ### Changed +- **CaDecon** raster overview uses the shared viridis colormap and drops the + intensity colorbar (activity is assumed to span 0→full; absolute values are + not meaningful) (PR #159) +- `calab-solver` tuning-constant hygiene: introduced `SeedConfig`, shared + `baseline::DEFAULT_BASELINE_QUANTILE`, and a named `BASELINE_EMA_WEIGHT`; + deduplicated the bi-exponential fast-component grid bounds so the grid search + and golden-section refinement cannot drift (no behavior change) (PR #163) +- Repo hygiene: ignore local Python virtualenvs (`.venv*/`) (PR #156) - Improved tutorial terminology and scientific accuracy (PR #59) -### Added +### Fixed -- Comprehensive README files across all packages and apps (PR #58) +- `calab-solver` FFI boundaries (WASM and PyO3) reject non-finite (NaN/Inf) + input traces with an explicit error instead of returning garbage results + (PR #161) +- Banded AR(2) forward model aligned via a one-sample source delay so the + reconvolution matches the double-exponential kernel (PR #157) ## [2.0.6] - 2026-02-19 diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 192cd44f..046129f8 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -21,16 +21,18 @@ npm run dev # Start dev server CaLab is an npm workspaces monorepo: -| Workspace | Path | Description | -| ------------------ | --------------------- | ----------------------------------------------- | -| `catune` | `apps/catune/` | SolidJS app — deconvolution parameter tuning | -| `carank` | `apps/carank/` | SolidJS app — CNMF trace quality ranking | -| `@calab/core` | `packages/core/` | Shared types, pure math, WASM adapter | -| `@calab/compute` | `packages/compute/` | Generic worker pool, warm-start cache | -| `@calab/io` | `packages/io/` | File parsers (.npy/.npz), validation, export | -| `@calab/community` | `packages/community/` | Supabase DAL, submission logic, field options | -| `@calab/tutorials` | `packages/tutorials/` | Tutorial type definitions, progress persistence | -| `@calab/ui` | `packages/ui/` | Shared layout: Shell, Panel, VizLayout | +| Workspace | Path | Description | +| ------------------ | --------------------- | ------------------------------------------------------------------------------ | +| `catune` | `apps/catune/` | SolidJS app — deconvolution parameter tuning | +| `carank` | `apps/carank/` | SolidJS app — CNMF trace quality ranking | +| `cadecon` | `apps/cadecon/` | SolidJS app — automated InDeCa deconvolution | +| `admin` | `apps/admin/` | SolidJS app — community-submission admin | +| `@calab/core` | `packages/core/` | Shared types, pure math, WASM adapter | +| `@calab/compute` | `packages/compute/` | Generic worker pool, warm-start cache | +| `@calab/io` | `packages/io/` | File parsers (.npy/.npz), validation, export | +| `@calab/community` | `packages/community/` | Supabase DAL, submission logic, field options | +| `@calab/tutorials` | `packages/tutorials/` | Tutorial type definitions, progress persistence | +| `@calab/ui` | `packages/ui/` | Shared layout (Shell, Panel, VizLayout) + chart primitives (`@calab/ui/chart`) | All packages are consumed as TypeScript source — Vite transpiles them directly via path aliases. No separate build step needed for development. @@ -58,6 +60,8 @@ You can also run scripts in a specific workspace: ```bash npm run dev -w apps/catune # Start CaTune dev server npm run dev -w apps/carank # Start CaRank dev server +npm run dev -w apps/cadecon # Start CaDecon dev server +npm run dev -w apps/admin # Start admin dev server npm run test -w apps/catune # Run app tests only npm run test -w packages/io # Run io package tests only ``` diff --git a/docs/NEW_APP.md b/docs/NEW_APP.md index 16e94bb5..b499d30a 100644 --- a/docs/NEW_APP.md +++ b/docs/NEW_APP.md @@ -61,7 +61,7 @@ npm auto-discovers the new workspace under `apps/*`. ```jsonc // package.json (root) "scripts": { - "typecheck": "tsc -b apps/catune apps/carank apps/" + "typecheck": "tsc -b apps/catune apps/carank apps/cadecon apps/admin apps/" } ``` diff --git a/packages/core/README.md b/packages/core/README.md index 36af770a..f6edd95d 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -16,16 +16,17 @@ Only `wasm-adapter.ts` may import from the WASM solver package (`crates/solver/p ## Exports -| Export | Source | Description | -| --------------------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------- | -| `initWasm`, `Solver` | `wasm-adapter.ts` | WASM initialization and solver class | -| `CaTuneExportSchema` | `schemas/export-schema.ts` | Valibot schema for JSON export validation | -| `NpyResult`, `NpzResult`, `ValidationResult`, `ImportStep`, ... | `types.ts` | Shared data types | -| `SAMPLING_RATE_PRESETS` | `types.ts` | Common sampling rate options | -| `CellSolverStatus`, `SolverParams`, `WarmStartStrategy`, ... | `solver-types.ts` | Worker communication protocol types | -| `computeAR2` | `ar2.ts` | AR(2) coefficient derivation from tau parameters | -| `PARAM_RANGES` | `param-config.ts` | Scientifically reasonable parameter ranges (rise 1–500 ms, decay 50–3000 ms, sparsity 0–10) | -| `formatDuration` | `format-utils.ts` | Duration formatting utility | -| `computePeakSNR`, `snrToQuality` | `metrics/snr.ts` | Peak signal-to-noise ratio and quality tier classification | -| `computeSparsityRatio`, `computeResidualRMS`, `computeRSquared` | `metrics/solver-metrics.ts` | Solver quality metrics | -| `computePeriodogram` | `spectrum/fft.ts` | Power spectral density computation | +| Export | Source | Description | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `initWasm`, `Solver`, `indeca_solve_trace`, `indeca_estimate_kernel`, `indeca_fit_biexponential`, `indeca_compute_upsample_factor`, `seed_trace`, `simulate_traces`, `get_simulation_presets` | `wasm-adapter.ts` | WASM init, the FISTA `Solver` class, and the InDeCa/seed/simulation entry points. `indeca_fit_biexponential` returns a bi-exp fit including a `fit_mode` outcome (`TwoComponent`/`SlowOnly`/`Degenerate`/`Empty`). | +| `CaTuneExportSchema` | `schemas/export-schema.ts` | Valibot schema for JSON export validation | +| `NpyResult`, `NpzResult`, `ValidationResult`, `ImportStep`, ... | `types.ts` | Shared data types | +| `SAMPLING_RATE_PRESETS` | `types.ts` | Common sampling rate options | +| `CellSolverStatus`, `SolverParams`, `WarmStartStrategy`, ... | `solver-types.ts` | Worker communication protocol types | +| `computeAR2` | `ar2.ts` | AR(2) coefficient derivation from tau parameters | +| `PARAM_RANGES` | `param-config.ts` | Scientifically reasonable parameter ranges (rise 1–500 ms, decay 50–3000 ms, sparsity 0–10) | +| `CONVERGENCE_RANGES` | `param-config.ts` | Shape-space convergence defaults: `convergenceTol` (0.02), `convergencePatience` (3), `convergenceMinIters` (2), `finalSelectionWindow` (5) | +| `formatDuration` | `format-utils.ts` | Duration formatting utility | +| `computePeakSNR`, `snrToQuality` | `metrics/snr.ts` | Peak signal-to-noise ratio and quality tier classification | +| `computeSparsityRatio`, `computeResidualRMS`, `computeRSquared` | `metrics/solver-metrics.ts` | Solver quality metrics | +| `computePeriodogram` | `spectrum/fft.ts` | Power spectral density computation | diff --git a/packages/ui/README.md b/packages/ui/README.md index 0dd8964e..bb5c616a 100644 --- a/packages/ui/README.md +++ b/packages/ui/README.md @@ -1,8 +1,8 @@ # @calab/ui -Shared SolidJS layout components and CSS design tokens for the CaLab monorepo. +Shared SolidJS layout components, chart primitives, and CSS design tokens for the CaLab monorepo. -Depends on `@calab/tutorials` (for TutorialPanel and TutorialLauncher components). External dependency: `solid-js`. +Depends on `@calab/tutorials` (for TutorialPanel and TutorialLauncher components). External dependencies: `solid-js`, `uplot`. ``` @calab/tutorials @@ -25,6 +25,19 @@ apps/catune, apps/carank | `TutorialLauncher` | `TutorialLauncher.tsx` | Tutorial launch button component | | `Card` | `Card.tsx` | Generic card wrapper component | +The table above covers the layout components; the barrel (`src/index.ts`) also re-exports the community/auth widgets and the chart utilities below. Consult `src/index.ts` for the authoritative export list. + +## Chart utilities (`@calab/ui/chart`) + +Shared uPlot primitives so every app charts consistently (also re-exported from the top-level barrel): + +| Export | Source | Description | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------- | +| `OKABE_ITO`, `OKABE_ITO_CYCLE`, `NEUTRAL`, `TRACE_COLORS`, `GROUND_TRUTH_COLORS`, `KERNEL_FIT_COLORS`, `METRIC_COLORS`, `DISTRIBUTION_COLORS`, `subsetColor`, `withOpacity` | `series-utils.ts` | Colorblind-safe Okabe-Ito palette and semantic color roles for series | +| `VIRIDIS_LUT`, `viridisRGB`, `viridisCss` | `colormap.ts` | 256-entry viridis lookup table + RGB/CSS accessors (raster heatmaps) | +| `niceTicks` | `chart-math.ts` | Human-friendly axis tick values (1/2/5×10ⁿ) | +| `chartAxis`, `labeledAxis`, `syncCursor`, `staticCursor`, `safeRange`, `integerTickValues`, `hiddenTickValues` | `axis-helpers.ts` | uPlot axis/cursor/range builders with shared theming | + ## CSS Layout styles are in `styles/layout.css`. Apps define their own design tokens (colors, spacing, shadows) in their `global.css` files, following the dark-theme scientific instrument aesthetic. diff --git a/python/README.md b/python/README.md index 5daf9785..10a6b72a 100644 --- a/python/README.md +++ b/python/README.md @@ -197,17 +197,19 @@ For full API documentation with parameter details, see [calab.readthedocs.io](ht ### CaDecon -| Function / Type | Description | -| -------------------------------------------------- | ---------------------------------------------------------------------------- | -| `decon(traces, fs, ...)` | Open CaDecon in browser | -| `HeadlessBrowser()` | Context manager for headless browser sessions | -| `solve_trace(trace, tau_rise, tau_decay, fs, ...)` | Single-trace InDeCa pipeline | -| `estimate_kernel(traces_flat, spikes_flat, ...)` | Free-form kernel estimation | -| `fit_biexponential(h_free, fs, ...)` | Bi-exponential kernel fit | -| `compute_upsample_factor(fs, target_fs)` | Upsample factor for target rate | -| `CaDeconResult` | Namedtuple: activity, alphas, baselines, pves, kernels, fs, metadata | -| `SolveTraceResult` | Namedtuple: s_counts, alpha, baseline, threshold, pve, iterations, converged | -| `BiexpFitResult` | Namedtuple: tau_rise, tau_decay, beta, residual, fast-component fields | +| Function / Type | Description | +| -------------------------------------------------- | -------------------------------------------------------------------------------- | +| `decon(traces, fs, ...)` | Open CaDecon in browser | +| `HeadlessBrowser()` | Context manager for headless browser sessions | +| `solve_trace(trace, tau_rise, tau_decay, fs, ...)` | Single-trace InDeCa pipeline | +| `estimate_kernel(traces_flat, spikes_flat, ...)` | Free-form kernel estimation | +| `fit_biexponential(h_free, fs, ...)` | Bi-exponential kernel fit | +| `compute_upsample_factor(fs, target_fs)` | Upsample factor for target rate | +| `CaDeconResult` | Namedtuple: activity, alphas, baselines, pves, kernels, fs, metadata | +| `SolveTraceResult` | Namedtuple: s_counts, alpha, baseline, threshold, pve, iterations, converged | +| `BiexpFitResult` | Namedtuple: tau_rise, tau_decay, beta, residual, fast-component fields, fit_mode | + +> **Non-finite input:** the deconvolution/fit entry points (`run_deconvolution*`, `solve_trace`, `estimate_kernel`, `fit_biexponential`, and the batch paths) raise `ValueError` if an input trace/array contains `NaN` or `Inf`, rather than returning garbage. `fit_biexponential`'s `fit_mode` reports the outcome (`"TwoComponent"` / `"SlowOnly"` / `"Degenerate"` / `"Empty"`). ### Shared Utilities diff --git a/python/docs/guides/cadecon.md b/python/docs/guides/cadecon.md index 0f9eff7d..eeb094d0 100644 --- a/python/docs/guides/cadecon.md +++ b/python/docs/guides/cadecon.md @@ -224,6 +224,8 @@ calab.solve_trace( Returns a `SolveTraceResult` namedtuple with fields: `s_counts`, `alpha`, `baseline`, `threshold`, `pve`, `iterations`, `converged`. +Raises `ValueError` if `trace` (or `warm_counts`) contains a non-finite value (`NaN` or `Inf`) — the FFI boundary rejects non-finite input rather than returning garbage. + ### `estimate_kernel()` Estimate a free-form kernel from traces and their corresponding spike trains. This is the "kernel step" of the InDeCa iteration. @@ -259,6 +261,8 @@ calab.estimate_kernel( Returns a float32 array of shape `(kernel_length,)` -- the estimated free-form kernel. +Raises `ValueError` if any input array contains a non-finite value (`NaN` or `Inf`). + ### `fit_biexponential()` Fit a parametric biexponential model to a free-form kernel. Optionally refines with a two-component (slow + fast) model. @@ -282,7 +286,9 @@ calab.fit_biexponential( | `skip` | Number of leading samples to skip in the fit. | | `warm` | Previous `BiexpFitResult` for warm-start. | -Returns a `BiexpFitResult` namedtuple with fields: `tau_rise`, `tau_decay`, `beta`, `residual`, `tau_rise_fast`, `tau_decay_fast`, `beta_fast`. Fast-component fields are 0 if a single-component fit was used. +Returns a `BiexpFitResult` namedtuple with fields: `tau_rise`, `tau_decay`, `beta`, `residual`, `tau_rise_fast`, `tau_decay_fast`, `beta_fast`, `fit_mode`. Fast-component fields are 0 if a single-component fit was used. `fit_mode` is a string reporting the fit outcome — one of `"TwoComponent"`, `"SlowOnly"`, `"Degenerate"` (no positive slow amplitude — untrustworthy), or `"Empty"` (no fit produced). + +Raises `ValueError` if `h_free` contains a non-finite value (`NaN` or `Inf`). ### `compute_upsample_factor()`