test(ingestion): e2e tests for all Task Delivery metrics (+ MV-refresh fix)#1498
Merged
mitasovr merged 14 commits intoJul 10, 2026
Merged
Conversation
Add declarative YAML e2e tests for four Task Delivery bullet metrics that derive from the status-interval path (task_status_intervals → task_dev_seconds_per_issue): task_dev_time, mean_time_to_resolution, pickup_time and flow_efficiency. All four share one bronze scenario — a single Jira issue (Alice) closed via Open → In Progress → Closed — so one set of intervals exercises every metric: 5d dev / 7d lead / 2d pickup → task_dev_time=120h, mttr=7d, pickup_time=2d, flow_efficiency=71.4%. Each file asserts only its own metric_key on the IC bullet (...0011), bronze-only, full dbt + jira-enrich path. `./e2e.sh test -k task_delivery` = 7 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (14)
💤 Files with no reviewable changes (13)
📝 WalkthroughWalkthroughAdds a ChangesTask Delivery Jira metric e2e coverage
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…esh fix
Cover the last four Task Delivery bullet metrics so all 11 now have an e2e test:
- estimation_accuracy — estimate vs spent (time-tracking fields via changelog)
- task_reopen_rate — reopen events, gated on >=5 closes
- stale_in_progress — open issues idle >14d (today() snapshot, wide window)
- worklog_logging_accuracy — logged seconds / in-progress seconds
Supporting changes:
- Add a bronze_jira.jira_worklogs placeholder (+ schema + template) so the pure
dbt model jira__task_worklogs (silver:class_task_worklogs) builds in the rig;
this is the worklog_logging_accuracy numerator.
- Add timeoriginalestimate/timespent to the jira field catalog template — these
fields only arrive via the changelog, never the issue snapshot.
- Fix a refresh-completion race in refresh_intermediates: the poll accepted a
Finished status left by the PRIOR test's refresh and returned before the
just-triggered SYSTEM REFRESH VIEW ran, so the bullet query could read a stale
task_issue_current_state / task_status_intervals MV (flaky None/wrong values).
CH 24.8 has no SYSTEM WAIT VIEW, so wait for refresh_count to advance instead.
`./e2e.sh test -k task_delivery` = 11 passed (deterministic across runs);
full `./e2e.sh test` = 56 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
mitasovr
changed the title
…dian
Three targeted specs closing coverage gaps the per-metric happy-path tests left:
- bugs_to_task_ratio (ratio-of-sums): two days with unequal task counts so the
correct 100*Σbugs/Σtasks (=40) differs from avg-of-daily-ratios (=62.5),
guarding the issue constructorfabric#433 §3.3 aggregation against a regression to averaging.
- task_reopen_rate (gate branch): <5 closes → query_ref suppresses the rate to
NULL even though a reopen exists.
- mean_time_to_resolution (median): three issues with lead times [10,2,1] so the
period reducer's median (2.0) differs from the mean (4.33), pinning
quantileExact(0.5) (odd count avoids the upper-middle ambiguity on even sets).
`./e2e.sh test -k task_delivery` = 14 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
ktursunov
approved these changes
Jun 29, 2026
…-80aa57 Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech> # Conflicts: # src/ingestion/tests/e2e/metrics/schemas/bronze_jira.jira_worklogs.yaml # src/ingestion/tests/e2e/metrics/task_delivery_bugs_to_task_ratio_ratiosum_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_estimation_accuracy_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_flow_efficiency_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_mean_time_to_resolution_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_mean_time_to_resolution_median_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_pickup_time_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_stale_in_progress_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_task_dev_time_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_task_reopen_rate_gate_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_task_reopen_rate_jira.test.yaml # src/ingestion/tests/e2e/metrics/task_delivery_worklog_logging_accuracy_jira.test.yaml
…-80aa57 Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech> # Conflicts: # src/ingestion/tests/e2e/metrics/templates/jira_task.yaml
… claude/happy-beaver-80aa57
… gate
Upstream added a no-unasserted-stat gate (every returned bullet stat must be
asserted) and moved all fixtures to a 5-member-department shape. Rework the 11
task_delivery specs I added so each seeds an Engineering department (alice..erin,
ranks 1..5) with a known per-person value ladder and asserts the full stat set
(value/median/range_min/range_max/p25/p75) in a custom window + an empty window.
Ladders (query = erin): counts {1,2,3,4,5}, ratios {0/20/40/60/80} or
{20..100}. For task_dev_time/mean_time_to_resolution/pickup_time range_max is the
cohort P95 (= max on a 5-point ladder).
Two behaviours worth noting, both captured by the fixtures:
- Interval-derived metrics (task_dev_time, pickup_time, flow_efficiency,
worklog_logging_accuracy, task_reopen_rate) use PAST close dates (2026-03),
because task_status_intervals drops any interval ending after now()+1day — a
future window (2026-12) silently empties the interval MV. Current-state
metrics (tasks_completed, mttr, estimation, stale, bugs) are unaffected.
- task_reopen_rate: a person with closes but ZERO reopens gets a NULL rate
(the query_ref numerator sumIf(value<0) is NULL over no rows, not 0), so they
drop out of the distribution — the cohort is the 4 people with reopens.
- jira__task_worklogs dedups bronze by LIMIT 1 BY _airbyte_raw_id, so the 5
worklog rows need distinct _airbyte_raw_id (the template's shared placeholder
collapsed them to one).
Delete task_delivery_mean_time_to_resolution_median_jira: the redesigned base
mttr spec now exercises the cohort median directly.
`./e2e.sh test -k task_delivery` = 14 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
ktursunov
approved these changes
Jul 9, 2026
This was referenced Jul 9, 2026
The Metric coverage gate fails on stale skips: estimation_accuracy, flow_efficiency, mean_time_to_resolution, pickup_time, stale_in_progress, task_dev_time, task_reopen_rate and worklog_logging_accuracy are now value-tested by the task_delivery_* fixtures added in this PR, so their SKIP_LIST entries must go. The five still-untested keys (avg_slip, on_time_delivery, overrun_ratio, scope_completion, scope_creep) stay. Verified locally: metric_coverage.py against the CI run's catalog_metrics.json artifact now exits 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
mitasovr
enabled auto-merge (squash)
July 9, 2026 17:09
mozhaev-dev
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds declarative YAML e2e tests for the eight previously-untested Task Delivery bullet metrics, so all 11 metrics in the group now have an e2e test (
tasks_completed/bugs_to_task_ratio/due_date_compliancewere already covered).Status-interval metrics (from
task_status_intervals → task_dev_seconds_per_issue):task_dev_timemean_time_to_resolutionpickup_timeflow_efficiency100·Σdev/Σlead)Remaining metrics:
estimation_accuracytask_reopen_rate100·Σreopen/Σclose)stale_in_progressworklog_logging_accuracyEach spec is bronze-only and runs the full dbt +
jira-enrich+ gold path, asserting just its ownmetric_keyon the IC Task Delivery bullet (…0011). Per the rig's composition rule, templates carry only invariant scaffolding; every field driving an assertion is inline.Supporting changes
bronze_jira.jira_worklogsplaceholder (+ schema + template): the pure dbt modeljira__task_worklogs(silver:class_task_worklogs) is theworklog_logging_accuracynumerator; seeding the bronze makesderive_selectorsbuild it.timeoriginalestimate/timespentfield-catalog entries: these arrive only via the changelog, never the issue snapshot, soestimation_accuracymust set them there.refresh_intermediates: the poll accepted aFinishedstatus left by the prior test's refresh and returned before the just-triggeredSYSTEM REFRESH VIEWran, so the bullet query could read a staletask_issue_current_state/task_status_intervalsMV (flakyNone/wrong values across the group). CH 24.8 has noSYSTEM WAIT VIEW, so it now waits forrefresh_countto advance.Note on
task_reopen_rate: a reopened-but-still-open issue does not register a reopen — the last status interval clamps tofinal_close_at, so the post-reopen interval is dropped. The fixture closes reopened issues again (close→reopen→close), which is also the realistic shape.Test
cd src/ingestion/tests/e2e./e2e.sh test -k task_delivery→ 11 passed (deterministic across repeated runs)./e2e.sh test(full suite) → 56 passed🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes