Honest benchmarks dt#1
Open
everest-an wants to merge 6 commits into
Open
Conversation
…port - Fix evaluate_selective_copy KV-cache bug that inflated MT-LNN's advantage to a spurious x42; with 5-seed fair eval the three architectures are broadly comparable on Selective Copy - Withdraw unsupported 125M PPL / LRA / 89.5%-collapse / TinyLlama-needle numbers from paper (EN + ZH), replace with reproducible local results - Real local WikiText-103 (~84M): MT-LNN 34% lower PPL at matched step budget - New benchmarks: multi_seed_sweep, state_tracking (parity), mqar, ett_forecasting, irregular_sampling, physionet_mortality - Architecture: per-step dt continuous-time support (MTLNNModel->Block->Layer ->resonance); byte-identical when dt=None; 5 new tests; 57 CPU + 42 CUDA pass - PhysioNet-2012 mortality: MT-LNN leads (AUROC 0.823 vs 0.797/0.777); dt-in-decay is task-dependent (decisive on synthetic, redundant on clinical) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove all "the previous version reported X / we withdraw Y" meta-narrative from both EN and ZH papers (abstract, LM section, Selective Copy, AVP, ablation, appendix). The paper now states the current, real results directly (including the honest negative ones) without recounting earlier mistakes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigation of MT-LNN's long-sequence collapse (needle recall drops to chance at L>=2048 while Transformer/LNN converge). - Add opt-in protofilament_timescales flag (config, default off): each of the P protofilaments owns one distinct tau (P channels) instead of every protofilament running all S scales (P*S). Zero effect when off; paper numbers unchanged. - benchmarks/diagnose_mtlnn_longseq.py: per-scale gradient monitor + use_scan ablation. Finding: gradients are healthy (no explode/vanish), and the collapse is NOT the pscan recurrence (use_scan on/off similar). - benchmarks/real_text_needle/: original real-text needle scripts (the clean testbed where baselines converge and MT-LNN collapses), path-fixed + route-B integrated + incremental CSV write. Decisive result (L=2048 real-text needle): legacy MT-LNN recall 0.0 (loss 4.17) vs route-B recall ~0.05 (loss 2.45). Route-B lowers training loss ~40% but does NOT restore needle recall -> the collapse is structural (fixed-capacity recurrent state cannot do precise 2048-step retrieval), not channel redundancy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…N+ZH) Route-B ablation result: giving each protofilament one distinct time-scale (P channels vs P*S) lowers training loss 4.17->2.45 but does NOT restore needle recall at L=2048 (stays at chance). With healthy gradients and use_scan invariance, this establishes the collapse is structural (fixed recurrent state vs precise long-range retrieval), not channel redundancy or an optimisation artefact. Also fixed a stale "125M" scope line and removed a leftover retraction sentence in the ZH limitations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The confirming seed showed route-B's loss benefit is NOT robust: seed0 loss 2.45, seed1 loss 4.21 (same as legacy). Corrected the paper from the single-seed "4.17->2.45 substantial improvement" to the honest 2-seed finding: route-B neither reliably lowers loss (2.45/4.21 bimodal) nor restores recall (0.031, chance) -- which STRENGTHENS the structural-collapse conclusion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single entry-point for any new session/Codex: current state, done items, in-progress SOTA experiment, blockers (main M1 cleanup needs human force-push), next steps, and the hard-won pitfalls (withdrawn numbers, single-seed traps, eval-path bugs, GPU zombies, structural collapse, O1!=M1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.