Latent Liquidity Quant

Bayesian systemic stress monitoring with a Differentiable Particle Filter and Sinkhorn OT resampling.

🌐 Interactive Dashboard  ·  📊 Results & Figures  ·  📐 Theory & Methods  ·  📄 Technical Paper

---

Executive Summary

Quick Evaluation Snapshot

• Problem: Detect systemic stress before market drawdowns
• Approach: Bayesian state-space model + Differentiable Particle Filter
• Key Result:
  • Sharpe: 0.78 vs 0.52 (SPY)
  • Max DD: -13% vs -55%
• Insight: Works for endogenous crises, fails (correctly) on exogenous shocks. Detects liquidity-driven drawdowns 2–8 weeks in advance.
• Use Case: Portfolio overlay / risk management (not alpha generation)

Why This Problem Is Hard

• Only ~9 crisis events → extreme class imbalance
• Signals must be strictly causal (no leakage)
• Most apparent predictors disappear out-of-sample
• Crisis dynamics differ (endogenous vs exogenous)

Key Insight

The model consistently predicts endogenous crises (liquidity-driven drawdowns) but fails on exogenous shocks (COVID, SVB, Brexit).

This is not a weakness — it reflects a structural truth:

Financial markets only embed advance signals for risks that build internally. External shocks have no observable precursor in market microstructure.

This explains both:

• Strong performance in GFC / 2018 / 2022
• No signal ahead of COVID

In One Paragraph

Markets don't directly reveal systemic stress — it manifests indirectly through volatility clustering, spread widening, and correlation breakdown. This project recovers a hidden liquidity stress factor $L_t$ in real time from six daily market observables using a Bayesian state-space model with Sequential Monte Carlo inference. Two filter architectures are compared: a Bootstrap Particle Filter (BPF) with fixed parameters, and a Differentiable Particle Filter (DPF) that learns parameters online via gradient descent through Sinkhorn optimal-transport resampling. The DPF's posterior states are fed into a logistic early-warning layer to produce calibrated 60-day crisis probabilities, which are then used as a position overlay on SPY to evaluate tradeable signal quality.

---

Signal Quality

The DPF outperforms the BPF on every primary warning metric across a 19-year out-of-sample sample (2007–2026, 4,526 trading days):

Metric	DPF	BPF	Notes
Daily ROC-AUC	0.682	0.670	Warning signal vs 30-day pre-crisis labels
PR-AUC	0.102	0.097	Primary metric at 5.4% base rate
Brier Score	0.212	0.218	Lower is better
Endogenous episode AUC	0.567	0.500	Pre-crisis vs calm 6 months prior
Signal-to-noise E[L|crisis] − E[L|calm]	2.67	2.21
Crisis/calm discrimination ratio	340×	54×	GFC=0.784, calm=0.002
Posterior 90% CI width	0.548	0.219	BPF tighter (fixed params)
Tail prob AUC P(L>2)	0.709	0.726	BPF wins on contemporaneous

5 of 9 labelled crisis episodes detected at 13–60 day lead times. 3 exogenous shocks (COVID-19, Rate Shock, SVB bank run) correctly produce no advance signal — financial microstructure contains no 30-day precursor for external catalysts.

Crisis	DPF Lead	BPF Lead	Type
GFC (2007–2009)	13d	26d	Endogenous
Eurozone I (2010)	58d	58d	Endogenous
Eurozone II (2011)	56d	60d	Endogenous
China devaluation (2015)	60d	60d	Endogenous
Brexit (2016)	60d	60d	Exogenous†
Q4 Selloff (2018)	—	—	Endogenous
COVID-19 (2020)	—	—	Exogenous ✓
Rate Shock (2022)	—	—	Endogenous
SVB collapse (2023)	37d	37d	Exogenous†

† Brexit: signal driven by concurrent Eurozone stress, not referendum outcome. SVB: unrealised Treasury losses visible in credit spreads ~5 weeks before the bank run.

---

Strategy PnL

The crisis probability is used as a continuous position overlay (no threshold optimization). Three variants are evaluated: linear derisking, SPY/TLT rotation, and a convex power mapping that reduces carry cost during low-signal periods.

Strategy	Ann. Return	Ann. Vol	Sharpe	Max DD	Calmar	Ann. Turnover
Buy & Hold SPY	10.3%	19.7%	0.52	−55.0%	0.19	—
DPF Overlay (linear)	6.3%	8.0%	0.78	−13.1%	0.48	193%
DPF Rotate (SPY/TLT)	9.5%	11.6%	0.82	−33.9%	0.28	193%
DPF Overlay (convex)	7.4%	10.2%	0.72	−21.4%	0.35	190%
BPF Overlay	5.9%	7.6%	0.78	−15.2%	0.39	386%

All weights shifted 1 day (strict causality). Transaction cost: 1bp one-way on absolute daily weight change.

Sharpe t-stat (Lo 2002): ~3.3 for DPF Overlay (linear) over 4,526 days. ⚠ Alpha is GFC-concentrated (2008–09 contributes +22.6% episode alpha). Excluding GFC substantially reduces t-stat. Interpret as a tail-hedge profile, not persistent alpha.

Per-Episode Alpha — DPF Overlay (linear) vs Buy & Hold

Episode	DPF Return	B&H Return	Alpha	IR	Type
GFC (2007–09)	−10.9%	−33.5%	+22.6%	0.32	Endogenous
Eurozone I	−3.6%	−6.0%	+2.3%	0.35	Endogenous
Eurozone II	−5.0%	−2.9%	−2.1%	−0.54	Endogenous
China	−2.0%	−5.4%	+3.3%	1.79	Endogenous
Brexit	+2.6%	+3.0%	−0.3%	— ‡	Exogenous
Q4 Selloff	−2.0%	−17.8%	+15.8%	3.59	Endogenous
COVID-19	−8.5%	−20.0%	+11.4% ♦	0.77	Exogenous
Rate Shock	−3.9%	−19.1%	+15.2%	0.81	Endogenous
SVB	+0.4%	+3.1%	−2.7%	— ‡	Exogenous

‡ Short episode (< 20 trading days): IR unreliable — tracking error inflated by fast recovery. ♦ COVID alpha is incidental: the DPF had no advance signal; the strategy was partially derisked for unrelated reasons.

Convex mapping (SPY weight = 1 − prob²) stays near 100% when crisis probability is below 0.3, reducing the 2013–2019 bull market carry cost by ~50% vs linear with negligible impact on crash protection.

Strategy Interpretation

This is not a standalone alpha strategy.

• Returns are lower than SPY due to defensive positioning
• Performance comes from drawdown avoidance, not timing rallies
• Alpha is highly concentrated in crisis periods (e.g. +22.6% in GFC)

👉 The correct use case is:

• Overlay on existing portfolios
• Dynamic de-risking
• Tail-risk hedging

Not:

• Directional return generation

Limitations

• Only 9 crisis episodes → limited statistical power
• Signal depends on credit + volatility structure (may degrade in new regimes)
• No macro / geopolitical inputs (purely market-implied)
• Transaction costs simplified (1bp linear model)

What I Learned

• Differentiable particle filters are feasible but numerically fragile (NaNs, loss spikes)
• Most “predictive signals” disappear once strict causality is enforced
• Crisis prediction is fundamentally class-imbalanced and episodic
• Backtests are misleading unless you decompose performance by regime

---

_{Summary dashboard: KPIs, crisis probability timeline, lead-time bars, filter comparison}

  
_{Left: GFC window — DPF and BPF forecast cones evolving in real time. Right: COVID — correct zero advance warning for exogenous shock.}

---

Model Architecture

Observables (6 daily series):
  r_SPY  r_TLT  r_HYG  VIX  HY Spread (OAS)  SPY/TLT rolling correlation
                        ↓
            Bayesian State-Space Filter
                        ↓
Latent states (3):
  L_t  [liquidity stress]    h_t  [log-volatility]    z_t  [correlation driver]
                        ↓
    Forward simulation × 60 days → calibrated crisis probability P_crisis(t)

State Transitions

$$L_{t+1} = \phi_L L_t + \beta_L e^{h_t/2} + \varepsilon_L$$ $$h_{t+1} = \mu_h + \phi_h(h_t - \mu_h) + \sigma_h!\left(\rho_\text{lev},\varepsilon_t^\text{ret} + \sqrt{1-\rho_\text{lev}^2},\xi_t\right)$$ $$z_{t+1} = \phi_z z_t + \gamma L_t + \varepsilon_z$$

Two structural improvements over the baseline SV model:

Stochastic Volatility with Leverage (SVL) — the $h_t$ transition uses the standardised prior return $\varepsilon_t^\text{ret} = r_t^\text{SPY} / (\sigma_\text{SPY} e^{h_t/2})$, capturing the empirical leverage effect ($\rho_\text{lev} \approx -0.73$). Volatility rises when prices fall. Validated: $h_\text{calm} \approx -0.6$, $h_\text{GFC} \approx +1.1$ → VIX ≈ 18 calm, VIX ≈ 30 at peak.

Diagonal-Rotation-Diagonal (DRD) covariance — replaces the restrictive equal-variance assumption. Each asset has its own baseline daily vol ($\sigma_\text{SPY}=1%$, $\sigma_\text{TLT}=0.8%$, $\sigma_\text{HYG}=0.6%$) scaled by the common regime $e^{h_t/2}$. Covariance: $\Sigma_t = D_t R(\rho_t) D_t$ where $\rho_t = \tanh(z_t)$.

DPF vs BPF

	BPF	DPF
Resampling	Multinomial	Sinkhorn OT ($\varepsilon=0.5$, 20 iters)
Parameters	Fixed (hand-calibrated)	Online gradient (Adam, $\eta=5\times10^{-4}$)
Particles	N=1000	N=500
Gradients	None	TF autodiff through resampling

Sinkhorn OT replaces stochastic discrete resampling with a smooth transport plan $P^\star(\mathbf{w})$, making $\partial \log p / \partial \bm\theta$ computable through the full filter. The DPF learns 10 transition parameters in a rolling 120-day causal window with loss-spike detection and NaN rollback (< 5% of windows affected).

Early-Warning Pipeline

The calibration layer solves a target-reframing problem: predicting the 30-day pre-crisis window, not the crisis itself.

Three rolling z-scored features (all strictly causal):

1. $f_1$ — 252-day z-scored $\hat{L}_t$ (normalises DPF's epoch-varying scale)
2. $f_2$ — rolling z-scored 5-day stress momentum $\Delta_5 f_1$ (fires earliest)
3. $f_3$ — equity drawdown depth

Fitted weights (DPF): $\beta_1=+0.77$ (level), $\beta_2=-0.30$ (momentum), $\beta_3=+0.57$ (drawdown). All signs are economically correct.

Predict → Calibrate → Predict double pass is intentional: the logistic model trains on first-pass posterior states; the second pass re-scores forward paths using the fitted model, eliminating circularity.

Neutral-baseline rescaling $P_\text{crisis} = \text{clip}((P_\text{raw} - p_0)/(1-p_0), 0, 1)$ produces clean [0,1] crisis probabilities rather than the saturated outputs from naive class-balanced logistic regression ($p_0 \approx 0.42$).

Forward simulation uses a separate fixed-parameter SSM — the DPF's adaptive window-specific parameters produce transient drift that inflates calm-period crisis probabilities when used for 60-day paths. Separating simulation from filtering keeps the long-run dynamics stable.

---

Quickstart

pip install -r requirements.txt
python data_pipeline/download.py
python data_pipeline/build_dataset.py

# Full eight-step pipeline
python main.py run

The pipeline executes automatically with gate checks between steps:

Step 1 / 8  —  Online DPF Parameter Training
Step 2 / 8  —  DPF Prediction   (first pass — heuristic scoring)
               ↳ gate: NaNs=0, Max L_t < 8
Step 3 / 8  —  DPF Calibration
Step 4 / 8  —  DPF Prediction   (second pass — calibrated scoring)
Step 5 / 8  —  BPF Prediction   (first pass — heuristic scoring)
               ↳ gate: NaNs=0, Max L_t < 8
Step 6 / 8  —  BPF Calibration
Step 7 / 8  —  BPF Prediction   (second pass — calibrated scoring)
Step 8 / 8  —  DPF vs BPF Benchmark  →  results/plots/comparison/

Individual commands:

python main.py train                              # online DPF learning
python main.py predict    --filter dpf            # filter + forward simulation
python main.py calibrate  --filter dpf            # fit logistic warning model
python main.py bench                              # 6 publication figures
python main.py pnl                                # strategy backtest
python main.py animate    --start 2007-06-01 --end 2009-06-01

---

Repository Structure

latent_liquidity_quant/
│
├── main.py                          # Unified CLI (train / predict / calibrate / bench / animate / pnl)
├── requirements.txt
│
├── src/
│   ├── models/
│   │   ├── finance_state_space.py   # NumPy BPF — SVL, DRD covariance, fixed params
│   │   └── tf_finance_ssm.py        # TF DPF — same model, differentiable, learned params
│   ├── filters/
│   │   ├── dpf_resampling.py        # Sinkhorn OT resampling (ε=0.5, 20 iterations)
│   │   └── sinkhorn.py              # Sinkhorn iteration
│   └── core/
│       └── crisis_labels.py         # 30-day pre-crisis labels, holdout logic
│
├── research/
│   ├── train_dpf_online.py          # Rolling-window online parameter learning
│   ├── dpf_crisis_predictor.py      # DPF filter + fixed-param forward simulation
│   ├── crisis_predictor.py          # BPF filter + forward simulation
│   ├── calibrate_crisis_score.py    # 3-feature logistic calibration
│   ├── compare_filters.py           # Full metrics + 6 publication figures
│   ├── pnl_analysis.py              # Strategy backtest: overlay, rotation, convex
│   ├── animate_comparison.py        # Dual-filter animated GIF
│   └── validate_calibration.py      # Reliability diagrams + Brier decomposition
│
├── data_pipeline/
│   ├── download.py                  # Yahoo Finance + FRED
│   ├── build_dataset.py             # Feature engineering + NPY export
│   └── features.py                  # Rolling correlation, drawdown, spread
│
├── results/
│   ├── trained_params/
│   │   └── dpf_learned_params.npz   # Full trajectory (10 params × 74 windows)
│   ├── dpf_crisis_res.npz           # DPF posteriors + calibrated crisis probabilities
│   ├── crisis_res.npz               # BPF posteriors + calibrated crisis probabilities
│   ├── crisis_calibration_dpf.pkl   # Fitted logistic pipeline (DPF)
│   ├── crisis_calibration_bootstrap.pkl
│   └── plots/
│       ├── comparison/              # figure_1–6 + GFC/COVID GIFs
│       ├── pnl/                     # PnL figures: cumulative returns, rolling Sharpe, episode IR
│       ├── dpf/                     # DPF diagnostics + animations
│       └── bootstrap/               # BPF reliability diagrams
│
├── paper/
│   └── latent_liquidity_quant.pdf   # Technical write-up (15 pages, full derivations)
│
└── tests/
    ├── test_model.py                # Transition equations, likelihoods
    ├── test_filters.py              # BPF/DPF weight normalisation, ESS
    ├── test_state_space.py          # SVL leverage direction, DRD PD constraint
    └── test_logic.py                # Crisis labels, calibration pipeline

---

Key Design Decisions

Why Sinkhorn OT over multinomial resampling? Multinomial resampling is a discrete stochastic operation — gradients cannot flow through it. Sinkhorn OT produces a smooth, differentiable transport plan $P^\star(\mathbf{w})$, enabling $\partial \log p / \partial \bm\theta$ to propagate through the full filter via autodiff.

Why the predict → calibrate → predict double pass? The logistic model trains on first-pass posterior states. Using those same states to evaluate the model creates a circularity — second-pass forward paths are scored with a freshly fitted model, avoiding in-sample contamination.

Why a separate fixed-parameter SSM for DPF forward simulation? The DPF's 120-day rolling window learns parameters specific to recent regimes. Applying those parameters to 60-day forward paths propagates transient noise, inflating calm-period probabilities. Separating estimation (adaptive) from simulation (fixed, stable long-run dynamics) is the architectural solution.

Why rolling z-score features for calibration? As DPF parameters adapt, the absolute scale of $L_t$ shifts across epochs. A static logistic boundary trained on one epoch fails on another. 252-day rolling z-scoring renders $L_t$ and its momentum stationary and cross-epoch comparable, allowing a single model to generalise over the full history.

Why the negative Brier Skill Score doesn't invalidate the model? At 5.4% base rate, $\text{Brier}_\text{clim} = 0.054 \times 0.946 = 0.051$. Any model firing above base-rate on pre-crisis days is penalised even when correct. BSS is an unreliable metric below 10% base rate; PR-AUC is the appropriate primary metric and is reported throughout.

Why no isotonic regression or Platt scaling? Post-hoc calibration requires $\geq 20$–30 independent episodes to generalise. With 7 training episodes and 2 held out, fitting a calibration curve would overfit to the training episodes with no reliable OOS evaluation. The neutral-baseline rescaling achieves the primary goal without this risk.

---

Outputs

File	Description
results/plots/comparison/figure_5_performance_dashboard.png	★ Start here — KPIs, timeline, lead-time bars, metric comparison
results/plots/comparison/figure_6_crisis_type_breakdown.png	Endogenous vs exogenous decomposition
results/plots/comparison/figure_1_stress_comparison.png	$L_t$ full series, DPF vs BPF, 90% CI
results/plots/pnl/pnl_cumulative_returns.png	Strategy PnL — cumulative returns, drawdown, position weights
results/plots/pnl/pnl_rolling_sharpe.png	Rolling Sharpe + cumulative alpha vs B&H
results/plots/pnl/pnl_episode_ir.png	Per-episode IR — linear vs convex, alpha bar chart
results/plots/comparison/gfc.gif	Animated dual-filter GFC window (2007–2009)
results/plots/comparison/covid.gif	Animated COVID window — correct zero advance warning
results/trained_params/dpf_learned_params.npz	Parameter trajectory: 10 params × 74 windows
paper/latent_liquidity_quant.pdf	Full technical write-up with derivations

---

Tests

pytest tests/ -v

Covers: SVL leverage sign, DRD positive-definiteness, filter weight normalisation, ESS calculation, crisis label construction, predict-calibrate-predict pipeline consistency, neutral-baseline rescaling.

---

Dependencies

numpy  pandas  scipy  scikit-learn  tensorflow  joblib
matplotlib  tqdm  yfinance  pandas-datareader

Python 3.9+. TensorFlow 2.x (CPU sufficient; GPU not required).

---

Reference

Full derivations and algorithm pseudocode: paper/latent_liquidity_quant.pdf

@misc{verma2026latentliquidity,
  author = {Verma, Amresh},
  title  = {Latent Liquidity Quantification: A Differentiable Particle Filter
             for Systemic Stress Monitoring and Crisis Early Warning},
  year   = {2026},
  url    = {https://github.com/meamresh/latent_liquidity_quant}
}

---

MIT License