Skip to content

fix: reduce_factors NaN aggregation silently corrupted other observables - #127

Merged
jc-macdonald merged 1 commit into
mainfrom
fix/119-reduce-factors-nan
Aug 24, 2026
Merged

fix: reduce_factors NaN aggregation silently corrupted other observables#127
jc-macdonald merged 1 commit into
mainfrom
fix/119-reduce-factors-nan

Conversation

@jc-macdonald

Copy link
Copy Markdown
Collaborator

Summary

Closes #119. reduce_factors() aggregated per-factor importance across observables via np.maximum, which propagates NaN (np.maximum(4.05, nan) == nan). A single observable that's legitimately undefined in some regimes (e.g. a Type-I error rate, only meaningful when the null hypothesis is true -- a completely reasonable, common Scorer pattern) silently erased a real, significant importance value found via a different observable for the same factor, dropping it from the output with no warning or error.

Where this surfaced

yoavram-lab/pp-eigentest's Phase 1a Morris screen had a factor (alpha) with a real, large importance (mu*=4.05, far above threshold) on mae_seq, silently excluded from reduce_factors' output because a different, NaN-valued observable (type_i_rate_*, NaN by design outside null regimes) happened to appear later in the importance dict's iteration order. Worked around locally there at the time (filtering NaN-containing observables before the call); this PR is the actual upstream fix.

Change

  • np.fmax (NaN-safe: ignores NaN in either operand) instead of np.maximum.
  • Seeded with NaN, not 0.0: a factor with zero valid (non-NaN) measurements across every observable now correctly stays NaN throughout rather than silently settling at a falsely-confident 0.0 (which would look identical to "tested and found unimportant").
  • Warns when that all-NaN case happens for one or more factors -- they're still dropped (no valid data to compare against threshold), but now visibly, since that's missing data, not confirmed unimportance.

Test plan

  • just ci passes, design.py at 99% coverage
  • New tests: a NaN-valued observable no longer erases a real signal found on a different observable (the exact pp-eigentest scenario); a factor NaN across every observable is dropped and warns; no false-positive warnings when there's no NaN at all

🤖 Generated with Claude Code

…les (#119)

reduce_factors aggregated per-factor importance across observables via
np.maximum, which propagates NaN (np.maximum(4.05, nan) == nan). A
single observable that's legitimately undefined in some regimes (e.g.
a Type-I rate, only meaningful when the null hypothesis is true --
a completely reasonable, common Scorer pattern) silently erased a
real, significant importance value found via a *different* observable
for the same factor, dropping it from reduce_factors' output with no
warning or error.

Found in practice: yoavram-lab/pp-eigentest's Phase 1a screen had a
factor (alpha) with a real, large Morris importance (mu*=4.05, far
above threshold) on one observable, silently excluded because a
different NaN-valued observable happened to appear later in the
importance dict's iteration order. Worked around locally there
(filtering NaN observables before the call); this is the actual fix.

Uses np.fmax (NaN-safe) instead, seeded with NaN rather than zero so a
factor with zero valid measurements across every observable stays NaN
throughout rather than settling at a falsely-confident 0.0 -- and warns
when that happens, since it reflects missing data, not confirmed
unimportance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jc-macdonald
jc-macdonald merged commit 9e16c35 into main Aug 24, 2026
4 checks passed
@jc-macdonald
jc-macdonald deleted the fix/119-reduce-factors-nan branch August 24, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reduce_factors silently corrupts all factors' importance when any observable is NaN

1 participant