fix: resolve all 28 findings from the refinement audit - #25
Merged
Conversation
added 6 commits
June 28, 2026 13:32
…ransport, guard pagination/fields, no-redirect, drift warning F2,F6: localize the interval-BEGINNING instant so DSTFlag disambiguates the repeated fall-back hour (validated on 2025-11-02: 25h day no longer collapses to 24h). F13: retry predicate skips non-retryable 4xx. F7: raise on data-without-fields. F15: keep paging when totalPages is absent. F16: coerce bad hourEnding. F25: follow_redirects=False. F18: warn on hub/LZ settlement-point allowlist drift.
… names F3: skip a commit whose payload matches the latest committed vintage (ignoring per-commit as_of/fetched_at provenance), so an unchanged hourly ERCOT re-pull no longer writes a fresh full-history vintage every run. This also makes the _commit_ercot per-symbol loop idempotent on retry (F9) and sharply cuts full-history rewrites (F12). F19: microsecond-resolution snapshot names avoid same-second collisions.
… partition-relative freshness F1: DAM schedule now fetches the NEXT operating day's curve (it clears midday), not a 2-day-stale settled day. F4: RT/load capture the CURRENT day intraday, not the previous one. ERCOT_DAILY is now keyed by the Central operating day (timezone + end_offset=2 so today/tomorrow are valid keys). F8: freshness is validated against the partition end, so backfilling an old operating day no longer fails the 2-business-day gate.
…ped CORS F5: optional X-API-Key gate (ENERGEX_READ_API_KEY) on data endpoints; healthz stays open; unauthenticated start logs a warning. F11: cap an unbounded /series read (413; bounded start/end reads exempt). F17: /curve returns 404 (not 500) on an unknown commodity. F27: opt-in origin-scoped CORS (ENERGEX_CORS_ORIGINS).
…gy + staleness coverage F10/F23/F26: load the repo-root .env from a package-relative path (no CWD/parent search) gated by an explicit ENERGEX_SKIP_DOTENV opt-out (set by conftest) instead of the 'pytest in sys.modules' heuristic; unifies precedence through os.environ. F21: remove the unused ercot_subscription_key_secondary field. F22: cover _POWER_PREFIX mode/library invariant + malformed-id rejection. F20: surface per-symbol staleness in the read-back check metadata.
…lowlist; drop legacy docs
F14: docker-compose now requires MINIO_ROOT_*/ARCTIC_*/NEO4J_AUTH via ${VAR:?...} instead of
baking world-known defaults (minioadmin / energex-arctic-secret) — a fresh deploy fails loudly
rather than silently using weak creds. F28: drop the blanket tests/.* gitleaks allowlist (it
masked nothing) so a real secret under tests/ is still caught. F24: relabel the removed legacy
:8000 service as the S2 read API and replace the dead ENERGEX_DB_PATH/INGEST_CRON env block with
the read-API config.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves all 28 adversarially-verified findings from the codebase audit, in six themed,
test-backed commits. +13 tests; full suite green, ruff + gitleaks clean.
Critical / High
curve (it clears midday); RT/load capture the CURRENT day intraday. ERCOT_DAILY is keyed by the
Central operating day (
timezone+end_offset=2). Regression test asserts the 14:00 CPT tickon day D selects D+1 (DAM) and D (RT/load).
collapsed (25h→24h) and
drop_duplicatesdeleted one. Now the interval beginning is localizedso
DSTFlagdisambiguates; validated against the real 2025-11-02 payload (100 distinct RTintervals preserved). Fall-back + spring-forward regression tests added.
commit_vintagenow skips a commit whose payloadmatches the latest vintage (ignoring per-commit
as_of/fetched_at), so an unchanged hourlyre-pull no longer writes a fresh full-history vintage; this also makes the per-symbol commit loop
idempotent on retry and sharply cuts O(N²) rewrites.
X-API-Keygate (ENERGEX_READ_API_KEY) on data endpoints;/healthzstays open; an unauthenticated start logs a warning.Medium
F7 raise on data-without-fields · F8 partition-relative freshness (backfills no longer fail the
gate) · F10/F23/F26 deterministic package-relative
.envload via an explicit opt-out · F11/seriesrow cap (413) · F13 retry only on 5xx/transport (no 4xx auth hammering) · F14 composerequires
MINIO_ROOT_*/ARCTIC_*/NEO4J_AUTHinstead of baking weak defaults · F19 microsecondsnapshot names.
Low
F15 keep paging when
totalPagesabsent · F16 coerce badhourEnding· F17/curve404 (not 500)· F18 hub/LZ allowlist drift warning · F20 per-symbol staleness surfaced in the read-back check ·
F21 drop dead config field · F22 symbology
_POWER_PREFIX+ malformed-id coverage · F24 relabellegacy :8000 docs as the S2 read API · F25
follow_redirects=False· F27 opt-in scoped CORS · F28narrow the gitleaks
tests/allowlist.Notes
creds, now provided via
.env. RotatingMINIO_ROOT_*/NEO4J_AUTHto strong values, and theARCTIC_*keys (which the existing MinIO data is locked to), requires a credential rotation +data migration — intentionally not done here.