[Issue #1069] Improve performance of dashboard queries via new analytics db indexes and views - #11819
Open
DavidDudas-Intuitial wants to merge 3 commits into
Open
Conversation
…exes The four burnup questions (Sprint_Burnup_by_issues/points, Deliverable_Burnup_Issue_Count/Points) were exact copies of their burndown counterparts, computing a single declining total_remaining series instead of exposing scope/completed as two independent series that both climb toward total scope. Renames the underlying CTEs (opened/closed -> scope/completed) and drops the subtraction. Also adds migration 0014, creating indexes on gh_issue(parent_issue_ghid) and the "latest row per deliverable/epic" lookups on gh_deliverable_quad_map/gh_epic_deliverable_map -- all three previously fell back to a full table scan or sort on every Deliverable_Burndowns/ Deliverable_Burnup/Deliverable_Data query. Verified via EXPLAIN ANALYZE against a copy of this schema that this switches those plans to index scans. This repo doesn't yet have seed/backup tooling for its analytics pipeline, so these are raw SQL/migration file changes only -- applying the updated question text to the live Metabase instance and running the migration still needs to happen manually. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DavidDudas-Intuitial
changed the title
DavidDudas-Intuitial
changed the title
DavidDudas-Intuitial
changed the title
Mirrors the optimization applied in delivery-intelligence (its proto- implementation lives here): the deliverable/epic burndown queries recompute a per-quad deliverable ranking and a latest epic-to-deliverable mapping via window functions/anti-joins and a DISTINCT ON over full history tables, plus a recursive epic/issue tree walk joined against gh_issue_history, all on every query. Migration 0015 adds mv_deliverables_per_quad, mv_latest_epic_deliverable_map, and mv_deliverable_daily_burndown to precompute these once. main.py's sync_data() refreshes them via a new refresh_materialized_views(db) function -- a plain function taking db as a parameter, matching the existing sync_projects/sync_quads/etc. pattern rather than living as a method on EtlDb. This does not update any of the 24 burndown/burnup/percent-done queries that could read from these views -- that's follow-up work once this repo has seed/backup tooling to manage the card SQL safely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…analytics-burndown-query-fixes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1069
Changes proposed
This PR:
1. Add indexes to improve performance of deliverable burndown/burnup queries
The deliverable burndown/burnup queries recursively walk the epic->issue tree via gh_issue.parent_issue_ghid, and look up the latest deliverable/epic mapping via gh_deliverable_quad_map/gh_epic_deliverable_map -- none of these had a supporting index, so each fell back to a full table scan or sort on every query execution.
To solve this, we add migration 0014_add_burndown_query_indexes.sql, creating indexes on gh_issue(parent_issue_ghid), gh_deliverable_quad_map(deliverable_id, d_effective), and gh_epic_deliverable_map(epic_id, d_effective).
2. Add materialized views to improve performance of deliverable burndown/burnup queries
The deliverable burndown/burnup queries each independently recompute a per-quad deliverable ranking and a latest epic-to-deliverable mapping, then walk a recursive epic/issue tree joined against
gh_issue_historyon every single query execution -- this is very expensive, and duplicated across 24 burndown/burnup-adjacent queries.To solve this, we create three materialized views in
0015_add_deliverable_epic_materialized_views.sql:mv_deliverables_per_quad-- per-quad deliverable rankingmv_latest_epic_deliverable_map-- latest epic-to-deliverable mappingmv_deliverable_daily_burndown-- deliverable-to-issue membership (epic tree walk + direct children) pre-joined againstgh_issue_historyand pre-aggregated by(deliverable_id, day)sync_data()inetldb/main.pynow refreshes all three via a newrefresh_materialized_views(db)function after every sync, so they're never more than one sync-cycle stale.3. Fix calculation bug in burnup queries
Due to a copy/paste bug,
155-Sprint_Burnup_by_points.sql,156-Sprint_Burnup_by_issues.sql,212-Deliverable_Burnup_Issue_Count.sql, and213-Deliverable_Burnup_Points.sqlall computed results wrongly (using burndown formula instead of burnup formula). Fixed the queries to use correct calculations.Context for reviewers
mv_deliverable_daily_burndowndepends onmv_latest_epic_deliverable_map, sorefresh_materialized_viewsrefreshes them in a fixed order to avoid recomputing against a stale epic-to-deliverable mapping.Validation steps
EXPLAIN ANALYZEthat the three affected joins/window functions switch fromSeq Scan+SorttoIndex Scanafter adding the equivalent indexes.analytics etl db_migrateagainst a fresh local Postgres instance -- migration 0015 applies cleanly on top of 0014, all three views created.analytics etl extract_transform_and_loadagainst the real HHS/simpler-grants-gov GitHub Project boards (3 projects, 69 deliverables, 62 sprints, 324 epics, 3603 issues) and confirmed all three views populate with correct, sensible data viarefresh_materialized_views.