feat: real ERCOT public-API ingestion (RT/DAM SPP + system load)

.env.example

@@ -52,11 +52,12 @@ FRED_API_KEY=your-fred-api-key-here
 # and needs no token; this is reserved for the keyed NOAA CDO API.
 # NOAA_TOKEN=your-noaa-cdo-token-here
 
-# ERCOT (reserved — connector not yet wired). The ERCOT public API needs BOTH an
-# Azure APIM subscription key AND OAuth (ROPC) username/password.
+# ERCOT public API (active). Azure AD B2C ROPC username/password PLUS an APIM
+# subscription key (the developer portal's "Primary key"). Secondary is optional.
 # ERCOT_USERNAME=your-ercot-username
 # ERCOT_PASSWORD=your-ercot-password
-# ERCOT_SUBSCRIPTION_KEY=your-ercot-subscription-key
+# ERCOT_API_KEY_PRIMARY=your-ercot-primary-subscription-key
+# ERCOT_API_KEY_SECONDARY=your-ercot-secondary-subscription-key
 
 # =============================================================================
 # Dagster (docker-compose only)

.gitleaks.toml

@@ -9,3 +9,8 @@ paths = [
   '''\.mypy_cache/.*''',
   '''\.ruff_cache/.*''',
 ]
+regexes = [
+  # ERCOT's PUBLIC OAuth client_id for the public-reports ROPC flow — published by ERCOT
+  # and required by every client; not a secret. Trips the generic-api-key entropy rule.
+  '''fec253ea-0d06-4272-a5e6-b478baeecd70''',
+]

README.md

@@ -6,9 +6,10 @@ A self-hosted, always-on energy-market data platform with a **bitemporal
 Energex is focused on **power markets**. It continuously ingests the EIA-930 Hourly
 Electric Grid Monitor (demand, day-ahead forecast, net generation, interchange, and
 generation-by-fuel for every US balancing authority), with NOAA weather and gas/oil
-fundamentals (EIA, FRED) as supporting context. An ERCOT nodal connector (RT + DA
-settlement point prices) is built and ships dormant until its OAuth credentials are
-supplied. Every batch is validated through a quality gate and committed to a versioned
+fundamentals (EIA, FRED) as supporting context. ERCOT nodal data is ingested live from
+the ERCOT public API: real-time and day-ahead settlement point prices for the trading
+hubs and load zones (`power.lmp` / `power.dalmp`) and ERCOT-wide actual system load
+(`power.load`). Every batch is validated through a quality gate and committed to a versioned
 ArcticDB store on MinIO. The store remembers not just *what* a value was, but *when each
 value became known*, so you can reconstruct exactly what the data looked like at any past
 moment.
@@ -93,7 +94,7 @@ Then open:
 - **Dagster UI** — http://localhost:3000 (assets, schedules, run history, backfills)
 - **MinIO console** — http://localhost:9001 (the ArcticDB object store)
 - **Neo4j browser** — http://localhost:7474
-- **Legacy API** — http://localhost:8000
+- **Read API (S2)** — http://localhost:8000 (`/series`, `/curve`, `/symbols`, `/libraries`, `/healthz`)
 
 Four schedules run by default and keep the store current with no manual intervention:
 EIA gas storage (Thursday), EIA petroleum status (Wednesday), FRED spot (weekday
@@ -127,7 +128,7 @@ Key variables (see [`.env.example`](.env.example) for the complete annotated lis
 | `MINIO_ROOT_USER`, `MINIO_ROOT_PASSWORD` | MinIO root (compose-only; provisions the scoped account) |
 | `ARCTIC_ACCESS_KEY`, `ARCTIC_SECRET_KEY` | Scoped service account created by `minio-init` |
 | `EIA_API_KEY`, `FRED_API_KEY`, `NOAA_TOKEN` | Source connector credentials |
-| `ERCOT_USERNAME`, `ERCOT_PASSWORD`, `ERCOT_SUBSCRIPTION_KEY` | ERCOT credentials (reserved) |
+| `ERCOT_USERNAME`, `ERCOT_PASSWORD`, `ERCOT_API_KEY_PRIMARY` | ERCOT public-API credentials (B2C username/password + APIM subscription key) |
 | `NEO4J_URI`, `NEO4J_USER`, `NEO4J_PASSWORD` | Entity graph |
 | `DAGSTER_PG_USERNAME`, `DAGSTER_PG_PASSWORD`, `DAGSTER_PG_DB` | Dagster Postgres (compose) |
 | `DEFAULT_LLM_PROVIDER`, `DEFAULT_LLM_MODEL`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OLLAMA_BASE_URL` | LLM provider (reserved, S3) |

docker-compose.yml

@@ -26,7 +26,7 @@ services:
       MINIO_SECRET_KEY: ${ARCTIC_SECRET_KEY:-energex-arctic-secret}
       TZ: UTC
     ports:
-      - "8001:8001"
+      - "8000:8001"
     depends_on:
       minio:
         condition: service_healthy

pyproject.toml

@@ -114,7 +114,7 @@ ignore = ["E501"]
 
 [tool.ruff.lint.per-file-ignores]
 "__init__.py" = ["F401"]
-"**/examples/*.py" = ["T201", "C408"]  # Allow print() and dict() in examples
+"src/examples/**" = ["T201", "C408"]  # Allow print() and dict() in demo scripts/notebooks
 "**/test_*.py" = ["T201"]  # Allow print() in test files
 
 [tool.black]

src/energex/__init__.py

@@ -47,6 +47,16 @@
 For more examples, see: src/examples/
 """
 
+# Load the repo-root .env for local runs so nested sub-configs (e.g. ConnectorConfig)
+# that read os.environ-only see credentials. Skipped under pytest to preserve the
+# offline tests' no-creds invariant; deployment injects env via docker-compose.
+import sys
+
+if "pytest" not in sys.modules:
+    from dotenv import load_dotenv
+
+    load_dotenv()
+
 # Pin ArcticDB's vendored AWS C SDK ahead of pyarrow's (phase-0 load-order hazard):
 # whichever of libarrow / arcticdb_ext loads first wins the AWS symbols, and if
 # pyarrow wins, ArcticDB's S3 client constructor aborts the process on macOS. The

src/energex/core/config.py

@@ -3,7 +3,7 @@
 from pathlib import Path
 from typing import Literal
 
-from pydantic import Field, SecretStr, field_validator
+from pydantic import AliasChoices, Field, SecretStr, field_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 from energex.core.exceptions import ConfigurationError
@@ -143,7 +143,11 @@ class ConnectorConfig(BaseSettings):
     ercot_username: str | None = Field(default=None, validation_alias="ERCOT_USERNAME")
     ercot_password: SecretStr | None = Field(default=None, validation_alias="ERCOT_PASSWORD")
     ercot_subscription_key: SecretStr | None = Field(
-        default=None, validation_alias="ERCOT_SUBSCRIPTION_KEY"
+        default=None,
+        validation_alias=AliasChoices("ERCOT_API_KEY_PRIMARY", "ERCOT_SUBSCRIPTION_KEY"),
+    )
+    ercot_subscription_key_secondary: SecretStr | None = Field(
+        default=None, validation_alias="ERCOT_API_KEY_SECONDARY"
     )
     noaa_token: SecretStr | None = Field(default=None, validation_alias="NOAA_TOKEN")