feat: real ERCOT public-API ingestion (RT/DAM SPP + system load) - #24
Merged
Conversation
added 14 commits
June 28, 2026 11:43
…schedules Replace the single placeholder ercot_spp asset with three real assets (ercot_rt_spp -> power.lmp, ercot_dam_spp -> power.dalmp, ercot_load -> power.load), each with a read-back quality-gate check and an enabled (RUNNING) schedule. Remove the temporary ErcotSppConnector alias.
…ct ignores to src/examples
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the placeholder ERCOT scaffold with a real, verified, complete ERCOT public-API
ingestion section. Every endpoint/auth/field fact was verified against the live ERCOT API.
What changed
client_id,scope,response_type=id_token, and send theid_tokenas Bearer(the scaffold wrongly read
access_token). Subscription key viaOcp-Apim-Subscription-Key.data(array-of-arrays) +fieldsenvelope andpaginate via
page/sizeover_meta.totalPages(the scaffold assumed array-of-objectsand had no pagination → silent truncation). Lowercase EMIL paths.
The scaffold parsed Central as UTC.
np6-905-cd) →power.lmp, DAM SPP (np4-190-cd) →new
power.dalmp, system load (np6-345-cd) →power.load. RT and DAM are split intoseparate libraries (same point/hour, different prices) to avoid a data collision.
server-side
settlementPointTypefilter, not all ~1,123 resource nodes.ERCOT_FUELMIXschema/route — ERCOT's public API has no fuel-mixreport; fuel mix is already served by EIA-930 (
EIA930.GEN_FUEL.ERCO).ERCOT_API_KEY_PRIMARY(with the legacyERCOT_SUBSCRIPTION_KEYas a fallback alias) + optional secondary.the freshness gate; fixtures are now relative to
date.today()..env.exampleupdated to reflect an active ERCOT section.Verification
src/energex+tests; Dagster definitionsvalidate via CLI.
intervals), DAM 312 rows, load 24 hourly rows — all pass the quality gate; timezone
conversion verified (CDT day → correct UTC range).
Follow-on fixes (pre-existing issues found during verification)
:8000(it was on:8001while a stale, orphaned legacy container squatted:8000)./series,/curve,/symbols,/libraries,/healthzare live; ERCOT data is queryable end-to-end (/series?library=power.lmp&symbol=hb_houston).src/examples(auto-fix + a scoped per-file-ignore for demoprint()/dict());ruff check src testsis now clean.import energexnow loads the repo-root.envfor local runs (guarded so pytest's no-creds invariant is preserved); the deployment still injects env via docker-compose.