improve change analysis: confirmed data echo, mandatory anomaly chart, remove stop-and-wait

native-skills/system/holistics-common/analyze-changes/SKILL.md

@@ -57,13 +57,23 @@ Brief:
 Parse the user's message for:
 - `metric` — the metric that changed (e.g., "revenue", "close rate", "daily signups")
 - `direction` — drop or increase
-- `time_reference` — when the change occurred (e.g., "in 2026", "last quarter", "since January")
+- `base_period` — the period where the change occurred (e.g., "May 2025")
+- `comparison_period` — the reference period (e.g., "April 2025")
 
 If `metric` is not clearly named, ask before proceeding:
 > "Which metric are you looking at?"
 
-Once the metric is confirmed, echo a one-line summary:
-> "Got it — you're seeing a [drop/increase] in **[metric]** [time_reference]."
+Fetch `current_value` (value in `base_period`) and `comparison_value` (value in `comparison_period`)
+from the dataset. Compute `delta` = current_value − comparison_value.
+
+Post an update to confirm the actual values:
+> "Got it — you're seeing a [drop/increase] in **[metric]** from **[comparison_value]**
+> ([comparison_period]) to **[current_value]** ([base_period]), [formatted delta]."
+
+Delta formatting:
+- Percentage metric (rate, %, ratio): "down/up **[|delta|] percentage points**"
+- Currency: "down/up **$[|delta|]** ([relative%]%)"
+- Count or other: "down/up **[|delta|]** ([relative%]%)"
 ```
 
 Execution: Do this yourself, no delegation.
@@ -74,7 +84,7 @@ Dependency: Step 1
 
 Brief:
 ```
-/detect_anomoly
+/detect_anomaly
 ```
 
 Execution:
@@ -86,7 +96,7 @@ Execution:
 
 ### Step 3 — Output the Stage 1 Summary Block
 
-Depdendency: Step 2
+Dependency: Step 2
 
 Brief:
 ```

native-skills/system/holistics-common/analyze-contribution/SKILL.md

@@ -163,7 +163,7 @@ Run only Steps 1 and 2, then proceed directly to Phase 3 using `user_specified_d
 
 **If `user_specified_dimensions` is empty — standard path:**
 
-Execute these five steps in the exact order below. Do not reorder, skip, or batch any step.
+Execute these four steps in the exact order below. Do not reorder, skip, or batch any step.
 
 **Step 1 — Output the overview sentence as plain text:**
 
@@ -175,65 +175,27 @@ Execute these five steps in the exact order below. Do not reorder, skip, or batc
 
 The table must include these columns: Comparison Value, Base Value, Delta Value, Pct Change.
 
-**Step 3 — Output the transition sentence as plain text:**
+**Step 3 — Post this update**
 
 > **There are multiple ways to explain what drove this change. In this analysis, I'll break it down by dimensions to identify which segments moved the most.** Here are my top [N] proposed dimensions:
 
-**Step 4 — Output the dimension list as plain text.** This is a text output step, not a tool call. Write the numbered list directly in your response. Each item must include a short reason — **5–8 words max, business-focused, no technical jargon**:
+**Step 4 — Output the dimension list as plain text, then proceed directly to Phase 3.** This is a text output step, not a tool call. Write the numbered list directly in your response. Each item must include a short, business-meaningful reason — **1 sentence max, focused on why this dimension is relevant to the metric change**:
 
 > **1. Dimension Name** — short business reason (e.g. "varies most by sales team", "differs by product line")
 >
 > **2. Dimension Name** — short business reason
 >
 > **3. Dimension Name** — short business reason
 
-**Step 5 — Output the Stage 1 Summary Block as plain text, then STOP:**
-
-> ---
-> Here's a summary of what we're going to analyze:
->
-> - **Metric**: [metric name]
-> - **Comparison period**: [comparison_period_literal] ([comparison_period_name])
-> - **Base period**: [base_period_literal] ([base_period_name])
-> - **Overall change**: [metric] [dropped/increased] [pct_change]%, from [comparison_value] to [base_value] (Δ [±overall_delta_value])
-> - **Dimension 1**: [dim1]
-> - **Dimension 2**: [dim2]
-> - **Dimension 3**: [dim3]
->
-> Shall I proceed to break down **[metric]** by each of these dimensions to see where the [drop/increase] happened the most?
-> ---
-
-**Do NOT call `ask_user` here.** Stop after outputting this block and wait for the user to reply in the chat.
-
----
-
-### Stage 2 Detection — check before Phase 3
-
-Before running Phase 3, check the conversation history:
-
-- **If a Stage 1 Summary Block is present** (contains confirmed metric, periods, overall delta, and dimension list):
-  - Check the user's most recent reply:
-    - **Affirmative** (e.g. "yes", "proceed", "looks good") → load `metric`, `base_period_literal`, `comparison_period_literal`, `overall_delta_value`, and confirmed dimension list from the block. Do NOT re-run Phases 1 or 2. Proceed directly to Phase 3 with dimension #1.
-    - **Requests changes** (e.g. "remove X", "swap Y for Z", "add Y", "use X instead") → update the dimension list based on the feedback, output a new Stage 1 Summary Block with the updated list, then proceed directly to Phase 3. Do not stop for another confirmation.
-    - **Negative** (e.g. "no", "stop", "that's enough", "skip it") → stop. Output a single closing sentence: "Got it — stopping here. Let me know if you'd like to explore any dimension further." Do not proceed to Phase 3.
-
-- **If no Stage 1 Summary Block is present**: this is a fresh invocation — run from Phase 1.
+After outputting the dimension list, proceed immediately to Phase 3. Do NOT stop or wait for user input.
 
 ---
 
 ### Phase 3 — Analyze one dimension (repeat for each confirmed dimension)
 
 **Plan (once, before starting any dimension analysis):**
 
-Post an update about the confirmed dimension list:
-> # Analyzing these dimensions
-> 1. [dim1]
-> 2. [dim2]
-> 3. [dim3]
-
-(Always show this update, even when the user accepted the proposed list without changes.)
-
-Then add new tasks to analyze the dimensions.
+Add new tasks to analyze the dimensions.
 
 NOTE: If you are going to delegate, make sure to reference `/analyze-contribution` in the brief, so that sub-agent can follow the prompt structures correctly.
 
@@ -274,7 +236,7 @@ Execute this once per dimension, in confirmed order. Complete the full phase —
 
 6. Compute `contribution_pct = delta_value / overall_delta_value × 100` for each segment using the scalar `overall_delta_value` from Phase 2.
 
-7. Compute and store `dimension_driver_pct = sum(delta_value for all driver_of_change segments) / overall_delta_value × 100` — used to rank dimensions in Phase 4.
+7. Compute and store `top_segment_contribution_pct = top_driver_segment_delta_value / overall_delta_value × 100` — the contribution of the single top driver segment; used in the Phase 4 "Contribution to change" column and to rank dimensions.
 
 **Show:**
 
@@ -342,15 +304,21 @@ Opening heading and summary line:
 >
 > [Metric] [dropped/increased] by [pct_change]% (from [comparison_value] to [base_value], [±abs(delta)]) between [comparison period] and [base period].
 
-Ranked table of all confirmed dimensions, sorted by `dimension_driver_pct` descending. Successfully analyzed dimensions appear first; failed dimensions appear at the bottom:
+Ranked table of all confirmed dimensions, sorted by `top_segment_contribution_pct` descending. Successfully analyzed dimensions appear first; failed dimensions appear at the bottom:
 
 | Dimension | Top Segment | Contribution to change |
 |-----------|-------------|------------------------|
 | … | | |
 
-For the **Contribution to change** column, apply the same threshold rule as the per-segment display:
-- If `|dimension_driver_pct| ≤ 100%`: show `[dimension_driver_pct]%`
-- If `|dimension_driver_pct| > 100%`: show `[N]× the net change` where [N] = round(|dimension_driver_pct| / 100, 1)
+For the **Contribution to change** column, show `top_segment_contribution_pct` — the contribution of the single top driver segment:
+
+> `top_segment_contribution_pct = top_driver_segment_delta_value / overall_delta_value × 100`
+>
+> **Critical:** The denominator is always `overall_delta_value` from Phase 2 (the metric's net change with no segment grouping). Never use the top segment's own delta as the denominator — that always produces 100%.
+
+Apply the threshold rule:
+- If `|top_segment_contribution_pct| ≤ 100%`: show `[top_segment_contribution_pct]%`
+- If `|top_segment_contribution_pct| > 100%`: show `[N]× the net change` where [N] = round(|top_segment_contribution_pct| / 100, 1)
 
 **Conditionally include (one line each):**
 

native-skills/system/holistics-common/detect-anomaly-aql/SKILL.md

@@ -6,70 +6,100 @@ user-invocable: false
 ---
 
 ### Method
-The expected band **follows the trend**. For each bucket `t` (`T` truncated to `<grain>`):
-- `actual` = `M`; `Δ` = the change from the previous bucket.
-- `expected` = previous actual **+ drift**, where drift = the trailing mean of `Δ` over the `W` buckets before `t`. This is the band's centre; it tilts with the trend.
-- `spread` = the trailing **stddev of `Δ`** over those `W` buckets.
-- `lower/upper_bound` = `expected ∓ k · spread`.
-- `z_score` = `(actual − expected) / spread` — is *this bucket's change* unusual versus the recent distribution of changes (stationary even when the level trends).
-- `is_anomaly` = `1` when `abs(z_score) > k`, else `0`. `k` default 3 (looser 2, stricter 4).
+Detect anomalies by working out what change to expect in each period from the periods before it, then flagging the periods whose actual lands far from that expectation.
 
-**Warm-up gate (required).** A bucket is only banded/flagged once it has a **full `W`-bucket window** of prior history. Partial-frame window stats return values (not null), so gate explicitly on `n_prior = window_count(...) ≥ W`: buckets with `n_prior < W` get **no band** and are **never flagged** — the leading `W` lead-in buckets are exactly these, shown as context.
+Everything is computed on **changes**, not levels: `Δ` = the change from the previous bucket. That is what lets the band follow a trend, since a steadily climbing series has large values but ordinary changes, and so never trips on the climb itself.
 
-Window `W` by grain — a whole number of cycles:
+For each bucket `t` (`T` truncated to `<grain>`), over **every bucket before it**:
 
-| Grain | `W` |
-|---|---|
-| day | 28 (4 weeks) |
-| week | 13 (a quarter) |
-| month | 12 (a year) |
-| quarter | 8 (two years) |
+- `avg_change` = mean of `Δ`. The change to expect.
+- `spread` = stddev of `Δ`. How much that change normally varies.
+- `expected` = previous `actual` + `avg_change`. Where this bucket should land.
+- `lower/upper_bound` = `expected` ∓ `k · spread`.
+- `z_score` = (`actual` − `expected`) / `spread`. How far off it landed, counted in `spread`s.
+- `anomaly_flag` = `1` when `abs(z_score) > k`, else `0`. `k` default 3 (looser 2, stricter 4). For the chart only.
+- `verdict` = `'unusual'`, `'normal'`, or `'not assessed'`. The one field the caller reads.
 
-If the series is shorter than `W + 1`, shrink `W` to about half the available history (minimum 4) and say so. With fewer than 8 points total, stop: *"Not enough history (n=X) to detect anomalies."*
+**There is no baseline length to set.** The frame is `..-1` — the first bucket of the series through the bucket before this one. Every bucket is judged against everything that came before it, so nothing has to be chosen per grain, per series, or per run.
 
-### Query
+**The opening buckets exclude themselves.** `spread` is a *sample* standard deviation, undefined for fewer than two values, so it comes back null until a bucket has at least two prior changes behind it — the fourth bucket of a series. `expected` is null before that too, since the first bucket has no previous value to project from. Nothing gates this: the nulls arise on their own. Those opening buckets are the **lead-in**: carried for context, never judged.
+
+**Nulls do all the branching.** `z_score` is arithmetic on `expected` and `spread`, so a null in either carries through, and `lower_bound` / `upper_bound` go null with it. A short series is therefore safe with no gate and no arithmetic: too little history returns nulls rather than an error. Read the result instead of predicting it; the caller reports what came back assessed.
+
+**`verdict` turns those nulls into a word, so the caller never reads a null.** Its three branches are tried in order, and the second one is what makes this work without a null test:
+
+- `abs(z_score) > k` → `'unusual'`
+- `abs(z_score) >= 0` → `'normal'` — true for every real number, and *not* true when `z_score` is null
+- otherwise → `'not assessed'`
 
-Placeholders: `M` = the metric measure (e.g. `gmv`), `T` = the bound time dimension (e.g. `bq_fct_order_items.created_date`), `<grain>` = the time grain, `W` = the window for the grain, `k` = the sensitivity threshold (default 3), `<dataset>` = the dataset name, `<metric label>` = a readable label.
+A null `z_score` fails both comparisons and falls to the `else`, so every row comes back carrying a literal word. Two things land in `'not assessed'`: a bucket with fewer than two prior changes behind it, and a perfectly flat stretch, where `spread` is `0` and `safe_divide` returns null. Both are correct — there is nothing to judge against in either case.
 
-`M`, `T`, `<grain>`, `W`, `k` are placeholders; the `-1..-1` range (previous bucket) is literal; every `window_*` call takes `order: T | <grain>() | asc(), partition: []`. Widen the timeframe filter by `W` buckets, and add any dimensional filters.
+**The baseline never forgets, and that is the cost.** With an expanding frame an unusual bucket raises `spread` for every bucket after it, permanently — there is no window for its influence to fall out of. One violent early spike can widen the band enough to hide milder anomalies for the rest of the series. The caller says so when the flagged set makes it relevant.
+
+### Query
+
+A complete query: monthly `total_revenue`, reporting on 2016 and 2017, `k` 3.
 
-Template:
 ```aql
-metric m_prev   = window_avg(M, -1..-1,  order: T | <grain>() | asc(), partition: []);
-metric m_delta  = M - m_prev;
-metric m_n      = window_count(M, -W..-1, order: T | <grain>() | asc(), partition: []);
-metric m_drift  = window_avg(m_delta, -W..-1,  order: T | <grain>() | asc(), partition: []);
-metric m_spread = window_stdev(m_delta, -W..-1, order: T | <grain>() | asc(), partition: []);
-metric m_expected = m_prev + m_drift;
-metric m_z      = safe_divide(M - m_expected, m_spread);
-metric m_lower  = case(when: m_n >= W, then: m_expected - k * m_spread, else: null);
-metric m_upper  = case(when: m_n >= W, then: m_expected + k * m_spread, else: null);
-metric m_anom   = case(when: and(m_n >= W, abs(m_z) > k), then: 1, else: 0);
+metric m_prev       = window_avg(total_revenue, -1..-1, order: orders.created_at | month() | asc(), partition: []);
+metric m_delta      = total_revenue - m_prev;
+metric m_avg_change = window_avg(m_delta, ..-1, order: orders.created_at | month() | asc(), partition: []);
+metric m_spread     = window_stdev(m_delta, ..-1, order: orders.created_at | month() | asc(), partition: []);
+metric m_expected   = m_prev + m_avg_change;
+metric m_z          = safe_divide(total_revenue - m_expected, m_spread);
+metric m_lower      = m_expected - 3 * m_spread;
+metric m_upper      = m_expected + 3 * m_spread;
+metric m_anom       = case(when: abs(m_z) > 3, then: 1, else: 0);
+metric m_verdict    = case(
+  when: abs(m_z) > 3,  then: 'unusual',
+  when: abs(m_z) >= 0, then: 'normal',
+  else: 'not assessed'
+);
 explore {
   dimensions {
-    bucket: T | <grain>()
+    bucket: orders.created_at | month()
   }
   measures {
-    actual: M,
+    actual: total_revenue,
     expected: m_expected,
     lower_bound: m_lower,
     upper_bound: m_upper,
     z_score: m_z,
-    is_anomaly: m_anom
+    anomaly_flag: m_anom,
+    verdict: m_verdict
   }
   filters {
-    // reporting window WIDENED by W buckets for warm-up; the @( … ) parentheses are REQUIRED.
-    T matches @(last <reporting + W> <grain>s)
-    // + any dimensional filters carried from the chart, e.g. bq_fct_order_items.merchant_country_code == "US"
+    orders.created_at < @2018-01-01
   }
   sorts {
     bucket asc nulls last
   }
 }
 ```
 
+Substitute `total_revenue` with `M`, `orders.created_at` with `T`, `month()` with the grain, and `3` with `k`. Leave both frames exactly as written: `..-1` is every bucket before this one, and `-1..-1` is the previous bucket.
+
+**The time filter has no lower bound.** Above, the report covers 2016 and 2017, but the filter carries no start date at all: the baseline is every prior bucket, so the fetch has to reach as far back as the data goes. Write it as an upper bound only — `orders.created_at < @2018-01-01` — and let the caller report on the range it was asked about.
+
+**Cropping the fetch changes the numbers, it does not just shorten them.** With an expanding baseline, every `expected`, `spread` and `z_score` depends on how far back the query reaches, so the same bucket scores differently under a narrower filter. Never crop to the range a chart happens to display, and never re-crop between the Step 2 query and the Step 3 chart — which is why the chart is drawn from this explore verbatim.
+
+**Dimensional filters go in the same block**, one per line:
+
+```aql
+  filters {
+    orders.created_at < @2018-01-01
+    orders.region == "West"
+  }
+```
+
 **Do not regress these:**
-- Count prior buckets with `window_count(M, -W..-1, …)` — **never** `window_sum(1, …)` (SQL-generation error).
-- The `n_prior ≥ W` gate is mandatory (partial windows return values, not null, and would false-flag the first `W` buckets).
-- `case(when: …, else: null)` is valid and is how the band is hidden in the lead-in.
-- The nested window (`window_stdev(m_delta, …)` where `m_delta` contains a window) compiles directly — one explore, no two-stage query.
+- The baseline frame is `..-1`, **not** a fixed count like `-12..-1`. This method has no baseline length; do not reintroduce one, and do not add an `n_prior` gate to compensate.
+- `window_stdev` is the **sample** standard deviation, and that is what makes the lead-in self-exclude: it is null below two values. `window_stdevp` (population) returns `0` there instead, and a zero spread scores every opening bucket as an extreme anomaly.
+- Nulls carry the lead-in through `m_expected` and `m_spread` into `m_z`, `m_lower`, `m_upper` and `m_anom`. Nothing downstream needs a guard of its own — do not add one.
+- **`verdict` and `anomaly_flag` are not redundant.** `verdict` is the caller's only input and carries three states; `anomaly_flag` exists solely so the chart has a numeric column series. Never drop one for the other, and never have the caller read `anomaly_flag`.
+- The nested window (`window_stdev(m_delta, …)` where `m_delta` contains a window) compiles directly: one explore, no two-stage query.
+- `anomaly_flag` must always return `0` or `1`, **never `null`**. The `else: 0` is mandatory: it is plotted as a column, and nulls cause rendering gaps instead of clean zeros. This is also why it cannot carry the third state and `verdict` exists.
+- `verdict`'s second branch must stay `abs(m_z) >= 0`, not a null test. It is what routes a null `z_score` to `'not assessed'` without any null-handling function.
+- `verdict` must never return null. Every row carries one of the three words.
+- **The fetch must reach the start of the series.** It is the only thing standing in for a baseline length now, and a bounded lower bound silently rescales every score in the result.
+- Dates are **literals, not calls**: `@2016-01-01`, `@2016-01-01 - 2017-12-31`, `@(last 12 months)`. There is no `@date(...)` function, and writing one is a syntax error at the `(`.