[truealpha-datahub] Accept full-denominator Yahoo persistence

[wangzitian0]

What changed

• Advances D3-staging-topt-capture from accepted E2 to E3 on the current canonical D2 data plane.
• Freezes 42 Yahoo call plans before interaction, covering the exact 21 TOPT instruments across original and changed vintages.
• Persists 42 immutable Yahoo raw rows and 42 normalized market-price rows, including canonical BRK.B/BRK-B mapping and Nasdaq/NYSE calendars.
• Replaces the 21 fixture market-price cells in each accepted D2 denominator while retaining its 63 identity, listing, and membership cells.
• Persists and replays two blocker-free 84-cell capture manifests and evaluations; identical reruns add no rows.
• Rolls back all database rows when any required symbol fails, and proves missing raw, normalized, confidence, or lineage evidence remains blocking.

Scope and claim ceiling

This PR accepts only D3 E3 Local/CI development evidence for 20 issuers, 21 instruments, 84 required cells, and two vintages. It does not activate Staging, register a schedule or release, prove live-source readiness, run strategy code, or make a Production claim. D3 E4 remains open for isolated Staging execution; wangzitian0/infra2#481 provides the deployment prerequisite.

Canonical manifest: governance/batches/D3-staging-topt-capture.v1.json revision 6
Manifest SHA-256: 3b7671af97dab8d8bbe4d8681319a6ad21c8ffa6fc402db6e64299cff4669965
Part of #171 and parent capability #27.

Validation

• Fresh database migration replay through db/roles.sql
• uv run pytest apps/data-engine/tests/batches/staging_topt_capture -q (49 passed, 1 skipped)
• uv run ruff check apps/data-engine/src/data_engine/batches/staging_topt_capture apps/data-engine/tests/batches/staging_topt_capture
• uv run mypy
• make issue-graph-check
• PR authorization with tools/check_delivery_governance.py --execute-acceptance
• Exact-head rung evidence: rung-evidence:D3-staging-topt-capture:10e96e7f7318a2f1832822282a6c539b018a3aaae3a41f843fffa492d2dd9ff2

[wangzitian0]

Integration note: #200 replaces direct release-images.yml push/PR behavior with a reusable exact-SHA build/publish gate. It already includes the data-engine image conditionally and makes it mandatory as soon as apps/data-engine/Dockerfile exists. When rebasing this PR after #200, please drop or reconcile its existing .github/workflows/release-images.yml edit rather than restoring the direct publication trigger.

[wangzitian0]

The branch rewrite removes the stale release-images.yml change entirely. This PR is now limited to the authorized D3 E3 evidence slice, so it does not restore or conflict with the exact-SHA publication gate from PR #200.

[Copilot]

Pull request overview

Advances the D3-staging-topt-capture delivery batch by introducing E3 “full-denominator” Yahoo interaction + persistence logic and corresponding integration tests, and updates the governance manifests/graph to reflect the new accepted rung and evidence hash.

Changes:

• Update governance tracking (vision-issue-graph.json) and the canonical D3 batch manifest to reflect accepted E3 evidence and target E4.
• Add a new D3 E3 implementation (e3_slice.py) that executes 42 Yahoo interactions across two vintages, persists raw + normalized rows, and emits two ready capture manifests/evaluations.
• Add E3 integration coverage (test_e3_slice.py) validating idempotence, full-denominator completeness, and rollback on a missing symbol; extend E2 normalization to allow overriding ticker/calendar identifiers.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
governance/vision-issue-graph.json	Updates the D3 batch node to target E4 and pin the new manifest hash.
governance/batches/D3-staging-topt-capture.v1.json	Bumps revision, updates base SHA, and records accepted E3 slice metrics while targeting E4.
apps/data-engine/tests/batches/staging_topt_capture/test_e3_slice.py	Adds end-to-end E3 tests for full-denominator persistence, idempotence, and rollback behavior.
apps/data-engine/src/data_engine/batches/staging_topt_capture/e3_slice.py	Implements D3 E3 full-denominator Yahoo interaction/persistence and manifest/evaluation production.
apps/data-engine/src/data_engine/batches/staging_topt_capture/e2_slice.py	Extends normalization to allow overriding ticker/calendar identifiers for persisted market-price payloads.
apps/data-engine/src/data_engine/batches/staging_topt_capture/e0_slice.py	Extends Yahoo DTO parsing to support NYSE (XNYS) metadata expectations alongside Nasdaq.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.