Context-Aware Distributionally Robust Deep Q-Learning (CARDQN)

A systematic, risk-managed portfolio optimization system. CARDQN extends the Distributionally Robust Deep Q-Learning (DRDQN) framework of Lu, Sester & Zhang (2025) to be less conservative without losing its robustness guarantee — recovering the original method's risk-adjusted performance while substantially raising return on the S&P 500.

Result (S&P 500, 1995–2024, with proportional transaction costs; CARDQN, 5-seed mean, 10-episode training): CARDQN reaches 3.40× terminal wealth — above the original authors' reported 2.89× — at even volatility, with higher Sharpe and Sortino and a comparable maximum drawdown.

The problem

DRDQN learns a trading policy that hedges against the worst-case market outcome inside an optimal-transport "ambiguity ball." This makes it robust to model error — but also over-conservative: it pools every market regime (bull, bear, calm, turbulent) into one transition model and hedges equally in all of them, leaving return on the table on a strongly drifting asset.

What CARDQN adds

Three core components, all computed from past data only (no forecasts, no look-ahead):

• Context tag (τ). A low-dimensional label of the current market regime — local trend, volatility band, proximity to recent extrema — distinguishing the conditions the agent is trading in.

• Fidelity score (φ). A per-regime confidence estimate of how reliably the transition model is identified. Validated out-of-sample (held-out transitions), so it rewards predictive accuracy, not in-sample fit — preventing overconfidence in regimes that merely look stable.

• Confidence-driven reference & radius. Both the reference measure and the ambiguity-ball radius tighten in well-identified, favorable regimes (deploy more capital where the model can be trusted) and relax back to the safe global model in uncertain or adverse ones (keep the hedge). Worst-case protection is preserved wherever it matters.

• Proposed (not yet tested): regime-conditional Bellman-target blend. A complementary rule that, in clearly favorable and well-identified regimes, blends the worst-case Bellman target toward the risk-neutral one by a confidence-gated, capped weight — relaxing the hedge where it is unlikely to be needed, while provably never becoming optimistic.

The net effect: the agent commits capital when and where it can trust its model, and stays defensive otherwise — materially less conservative, still robust.

Results

Agent	Terminal wealth	Volatility	Sharpe	Sortino	Max drawdown
S&P 500 buy-and-hold	9.52	0.19	0.41	0.57	−0.57
Lu et al. (2025), reported	2.89	0.12	0.27	0.37	−0.37
CARDQN (this work)	3.40	0.13	0.30	0.41	−0.39
RDQN, our reproduction (same budget)	1.59	0.11	0.13	0.18	−0.38

The headline comparison is CARDQN against the original authors' reported result: CARDQN delivers more terminal wealth (3.40× vs 2.89×) at even volatility, with higher Sharpe and Sortino and a comparable drawdown. We also independently reproduced the authors' RDQN through the CARDQN code path (context-aware layer off) at the same 5-seed, 10-episode budget — a same-codebase control (1.59×) that confirms the gain comes from the context-aware machinery, not the re-implementation. As with the original, CARDQN does not beat buy-and-hold on raw wealth, but it delivers a strongly improved risk-adjusted profile.

Engineering

• GPU-vectorized inner solver. The worst-case dual ascent is a single batched tensor (not a per-sample loop) — about 3.7× faster on GPU.
• Train on a simulator, evaluate on reality. Agents train on a signature-MMD generative market model and are backtested out-of-sample on the real S&P 500.
• Reproducible multi-seed campaigns on a university GPU cluster (SLURM); all hyperparameters in YAML; full results in results/.
• Clean Python package (rdqn/), no install required; gymnasium environments; TensorBoard logging.

Quick start

Python 3.10+, from the repo root:

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt   # GPU: use the cu118 wheel from pytorch.org
.venv/bin/python scripts/train_rdqn.py --smoke                        # functional check, seconds
.venv/bin/python scripts/train_rdqn.py --context-aware --episodes 10  # train CARDQN
.venv/bin/python scripts/evaluate.py --checkpoint runs/rdqn_agent_q.pkl  # backtest on the S&P 500

The 4-way ablation (off / uniform / radius / full = CARDQN), multi-seed and β-sweep: scripts/ablation.py. GPU periodic-eval probe (the campaign runner): scripts/probe_rdqn.py.

Repository layout

rdqn/            library package
  agents/        DQN + robust RDQN (CARDQN context-aware path)
  context/       context tag, fidelity, adaptive radius, blended reference
  solver/        Sinkhorn-dual robust Bellman solver (vectorized)
  envs/          MMD portfolio simulator
  evaluation/    S&P 500 backtest + risk metrics
config/          hyperparameters (experiments/*.yaml, incl. the context_aware block)
scripts/         train_rdqn, ablation, probe_rdqn, evaluate, stress_test
data/            spx.csv + pretrained MMD generator
docs/            CARDQN, Locality-Aware and Proceedings papers
results/         campaign results (results.csv + per-seed JSONs)

Papers & docs

• Context-Aware Distributionally Robust Deep Q-Learning (CARDQN) (2026) — paper (PDF) · also in-repo at docs/papers/CARDQN/.
• Proceedings on DRDQN and Locality-Aware DRDQN — docs/papers/.
• Lu, Sester, Zhang, Distributionally Robust Deep Q-Learning, arXiv:2505.19058 (the original method).

Disclaimer

CARDQN is research software for reproducibility and education, provided as-is, with no warranty (see the LICENSE). It is not investment advice and must not be used to make real trading decisions.

Meta

giuliocsr

CARDQN by giuliocsr is licensed under CC BY-NC-SA 4.0

https://github.com/giuliocsr