From 68ade36fe6faee3c799e562ba2f84081d8a36439 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 13:41:11 -0400
Subject: [PATCH 01/10] feat(newtab): Add hnt-backend-investigation skill

---
 plugins/newtab/README.md                      |   8 +
 .../skills/hnt-backend-investigation/SKILL.md | 312 ++++++++++++++++++
 .../references/data-sources.md                | 262 +++++++++++++++
 .../references/failure-modes.md               | 146 ++++++++
 4 files changed, 728 insertions(+)
 create mode 100644 plugins/newtab/skills/hnt-backend-investigation/SKILL.md
 create mode 100644 plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
 create mode 100644 plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md

diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md
index 407457f..e13bd86 100644
--- a/plugins/newtab/README.md
+++ b/plugins/newtab/README.md
@@ -9,3 +9,11 @@ Skills and tools for Firefox newtab development.
 Adds `@nova-cleanup` comments to Firefox newtab code for Project Nova parallel implementation tracking.
 
 Use this skill when adding cleanup comments to Nova-related changes. See [`skills/nova-cleanup-comments/SKILL.md`](skills/nova-cleanup-comments/SKILL.md) for details.
+
+### `hnt-backend-investigation`
+
+Diagnoses Home New Tab **backend** errors, outages, and data-quality defects — Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline.
+
+Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. It confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, and writes an evidence-backed `FINDINGS.md` naming the check that would have caught the problem earlier. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details.
+
+**Needs:** `gh`; `gcloud`/`bq` with a billing project; the `aws` CLI with a read-only SSO profile; a MySQL client plus Mozilla VPN; `uv` or `python3`; the Sentry MCP server if you have it. The skill runs on whatever subset you have — anything unreachable is raised as a task and reported under "Could not measure".
diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
new file mode 100644
index 0000000..a4437b7
--- /dev/null
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -0,0 +1,312 @@
+---
+name: hnt-backend-investigation
+description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc naming the check that would have caught it. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale.
+---
+
+# New Tab (HNT) Backend Investigation
+
+Diagnose a problem in the services behind Firefox Home New Tab recommendations — Merino, the
+curated corpus, admin-api, the article crawler, the ML section pipeline, the data pipelines. Not
+for in-tree `browser/extensions/newtab` frontend bugs.
+
+Your product is a **diagnosis**: a root cause, the evidence for it, the impact quantified, and the
+check that would have caught it earlier. Two populations can be affected, and every finding should
+say which: **Firefox New Tab clients** (everyone who sees recommendations) and **editors/curators**
+(the internal editorial team working through curation-admin-tools and admin-api).
+
+**How to reach each system, and the traps that will silently give you wrong answers** — read before
+your first probe in step 1: [references/data-sources.md](references/data-sources.md)
+
+**How these systems fail, the invariants worth asserting, and falsification moves that work** — read
+before you enumerate hypotheses in step 3, because it will change which ones you take seriously:
+[references/failure-modes.md](references/failure-modes.md)
+
+## How to think about this
+
+Four habits do most of the work. They matter more than any specific query below.
+
+**Confirm the symptom before explaining it.** Independently reproduce or measure the reported
+problem first, whatever its source. An alert can be miscalibrated, mis-scoped, or watching the
+wrong layer, and a description passed through two people can drift. Until you have seen the
+symptom yourself in data you pulled, you do not know what you are diagnosing.
+
+**Form the hypothesis before you run the check.** Write down what you expect to see, and what you
+would expect to see if the hypothesis were false. A query designed to confirm and a query designed
+to discriminate look similar and are not. The characteristic failure in this domain is a correct
+query paired with a wrong inference, stated confidently.
+
+**Work several lines at once.** When more than one explanation is plausible, probe them in parallel
+rather than following the most attractive one to its end. Breadth is cheap in the unmetered planes —
+Sentry, repo reads, CloudWatch alarm history, one live Merino request — and tunnel vision is the
+expensive failure. It is not free in BigQuery or Zyte: dry-run and select narrow columns before you
+fan out. Keep a list of every line you considered and record which you dropped and why.
+
+**Prefer measuring to reasoning.** Reproduce the request. Run the model on real inputs. Count the
+actual rows. Read the code path instead of assuming its behaviour. A plausible mechanism becomes a
+number, and only numbers survive review.
+
+Write like the evidence is asymmetric, because it usually is. Avoid "always", "never", "in every
+case"; say what each count is a count of, and whether it is a floor, a ceiling, or a sample.
+
+## Don't stall on the developer
+
+Every pause costs wall-clock, because the developer may not be watching. Default to proceeding:
+pick the sensible option, say which one you picked, and keep going. Ask a blocking question only
+when proceeding on any assumption would waste the whole investigation.
+
+When a source you cannot reach looks promising, raise it **once** as a pending task carrying the
+whole errand in its own text, prefixed `DEV:` so it reads as theirs rather than yours — e.g.
+`DEV: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at all`.
+Give the exact command, not a description of the problem; for an interactive login use the `!`
+in-session form so the output lands here, e.g.
+`! aws --profile <profile-you-found-in-~/.aws/config> sso login`. If the answer lives in a dashboard
+or console you have no credential for, do not ask for access at all — name the view, the filter, and
+the single number or shape you need, and ask them to read it back.
+
+**Then keep investigating the other hypotheses immediately. Do not wait.**
+
+Restate it exactly once more — whichever comes first: they confirm, the unblocked lines run out, or
+you are three probes into a line you had already judged weaker than the blocked one. That is the
+second and last ask. If there is still no response, finish steps 7 and 8 with `Status: blocked`,
+name the one unblock under "Could not measure", and leave the access task pending.
+
+Stopping outright is a last resort: only when the blocked source is the only thing that can settle
+the question **and** it cannot be reduced to a question they can answer for you — a dashboard,
+console, or explore is almost never a real stop. A request that quietly disappears is worse than one
+that was never made.
+
+## Workspace
+
+One directory holds the whole investigation: `~/investigation/<YYYY>-<mon><DD>-<slug>`, e.g.
+`~/investigation/2026-jul29-empty-de-sections`. Reuse a directory only when its FINDINGS.md is about
+the same symptom. Say which directory you chose; do not ask.
+
+- `FINDINGS.md` — one living document. Create it from the step-7 skeleton **before your first
+  probe**; steps 2, 4 and 6 write into it as they go, so step 7 is a final pass over a document that
+  already exists. Rewrite it in place, not as `FINDINGS-v2.md`.
+- `queries/` and `results/` — every query as a file, its output alongside under the same basename.
+- `api_responses/` — raw JSON from live calls. Save them even when they look boring; metered APIs
+  cost money to re-hit and the data may be gone tomorrow.
+- Aggregate before saving. Counts, rates, and shapes answer nearly every question these
+  investigations ask; this directory sits outside any repo and gets linked into tickets, so prefer a
+  distribution over a dump of rows carrying editor identities or user data.
+- Rewrite only the FINDINGS.md you wrote and delete only files this session created — anything else
+  in the directory belongs to another investigation.
+- Python work goes in an isolated venv inside this directory: `uv venv` if `uv` is installed,
+  otherwise `python3 -m venv`.
+
+## Step 1 — pin down the report
+
+Establish these five facts. Derive what you can — the alert or issue carries the symptom and its
+volume, stratifying the data gives you the surface and locale, a live request tells you whether it
+is still happening. Ask only for what is genuinely undiscoverable, in one batch, and keep working
+while you wait.
+
+| Fact | Why it matters |
+|---|---|
+| Exact symptom, verbatim | "Empty section" and "wrong items in section" have disjoint causes |
+| Who noticed, and how | Alert / editor report / spotted by hand — sets what evidence exists |
+| First and last seen, **with timezone** | Anchors the timeline; reports usually arrive late |
+| Surface, locale, section, client version | The stratum is very often the diagnosis |
+| Still happening right now? | Live reproduction vs. historical forensics |
+
+Then confirm it yourself.
+
+**If it is still happening, capture the perishable evidence first** — a live
+`curated-recommendations` request, current logs, current per-section counts — and save the raw
+responses under `api_responses/`. Build the step-2 prior while those probes are in flight; history
+keeps, live signal does not.
+
+**An error-rate signal is not yet a symptom.** Measure the erroring stage's *output* to fix the
+blast radius — items per section and freshest item timestamp per surface for assembly, a live Merino
+request for anything client-facing — so you can say whether Firefox clients are affected at all. An
+error floor that never reaches the output is a not-incident; errors flat with output at zero is
+worse than the alert says. When the report is a Sentry alert, resolve it to a concrete error and a
+volume, and decompose the issue by error message before trusting its title or its trend — the
+alert-quality checks are in `references/failure-modes.md`.
+
+**If you cannot reproduce it, that is a result, not a blocker.** Record the exact attempt — surface,
+locale, time, request, what you saw instead — then switch the question to why the reporter saw it
+and you do not: a stratum you did not hit, a window that has closed, a cache, a client version, or a
+monitor measuring something other than what it claims. Classify as `not-incident` only after that
+second question has an answer; unconfirmed is not refuted.
+
+A report relayed from an editor is ambiguous between what editors see in curation-admin-tools and
+what clients see on the surface. Probe both in parallel and state which population you confirmed.
+
+The moment you confirm client-visible impact that is still happening, state it in one line — what,
+how big, since when — and keep investigating.
+
+## Step 2 — establish what "normal" is
+
+You do not know what is currently normal in this system, so you have no basis on which to reject
+your own inference. Build that prior yourself before asking for it — most of it is in reach:
+
+- **Which surfaces are serving** — surfaces present and not disabled in the section data, or a live
+  Merino request for the locale in question.
+- **Whether the crawl is feeding them** — per-surface, per-source volume over the last day from the
+  crawl data. These two sets differ, and a crawled surface with no sections is not automatically a
+  bug.
+- **What shipped recently** — recent commits in the service repos, releases in Sentry, timestamps on
+  model or config artifacts.
+- **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of
+  week. Do not ask; compute it.
+- **Whether it has happened before** — search Sentry for the same signature, and the service repos'
+  GitHub issues.
+
+What is left is genuinely in the developer's head, and it is worth one short, non-blocking ask:
+anything in flight that would not show up in a repo or a dashboard, and whether a postmortem or
+incident register exists for this area. An existing write-up can answer in a paragraph what costs
+hours to reconstruct, so ask early — then carry on without waiting.
+
+Record what you derive and what they tell you in FINDINGS.md, keeping the two apart. When a later
+query contradicts the prior, treat your own inference as the suspect first, and say so.
+
+## Step 3 — enumerate hypotheses, then probe them in parallel
+
+List every plausible explanation before testing any of them. Widen the list from
+`references/failure-modes.md` — its failure-class tells cover classes the symptom table below does
+not, and "Why these failures are silent" names the mechanisms that make a broken pipeline return a
+valid-looking empty result rather than an error.
+
+| Symptom | Likely layer | Probe |
+|---|---|---|
+| Recommendations 500 / timeout / stale | Merino, or its upstream corpus API | Sentry + live `curated-recommendations` request |
+| Section empty or short | ML section assembly → corpus → Merino | Active items per section; freshest item timestamp per section |
+| Section missing entirely | Section config / section manager | Section rows for the surface; section-manager errors |
+| Wrong / stale / duplicate items | Corpus content or ranking | Trace one item end to end by `corpusItemId` |
+| Bad title, image, author, date | Crawl + hydration | Compare the stored item against a live re-extraction |
+| Whole publisher or domain absent | Crawl discovery / domain block | Per-domain counts over time, then the vendor's own status data |
+| Errors up, no reported user impact | The erroring stage itself | Assert that stage's output invariant downstream, then decompose the errors |
+| Editor-facing error or lockout | admin-api → corpus API | Sentry for the editor-facing services; reproduce the mutation |
+| Metrics or dashboard wrong | ETL / derived tables | Compare the derived table against its source |
+
+Before firing, write beside each hypothesis the result that would kill it. Then issue the probes as
+independent tool calls in a single batch, backgrounding anything slow and handing a line that needs
+several dependent steps to a subagent so the batch still returns together. Cap the first wave at
+four or five cheap, independent probes, and hold metered calls and large scans for the second wave.
+Name each `queries/` and `results/` pair after its hypothesis, so a result cannot be attributed to
+the wrong line.
+
+Two moves consistently break cases open: **leave the tool you started in** — when a source stops
+yielding, measure the same thing somewhere else, since the failure is often invisible in the plane
+you began with — and **compare against sibling strata**, which localises a fault faster than reading
+code. Both are expanded in `references/failure-modes.md`.
+
+## Step 4 — build the timeline
+
+An explicit `timestamp (UTC) | observation | source` table.
+
+- Normalise every timestamp to UTC and label it. Timezone mismatch is a common source of phantom
+  gaps, and a subtraction error can manufacture an outage that never happened.
+- Extend the window to weeks, not hours — these failures are frequently much older than the report.
+  Widen freely on the cheap planes; on a billed plane, widen only after a narrow window has shown
+  you something.
+- Distinguish *first occurrence* from *first noticed*, and state both.
+- Line the window up against the deploys and config changes from step 2.
+- Treat retention limits as limits: a "first seen" date can be the edge of a retention window rather
+  than onset. See `references/data-sources.md`.
+
+## Step 5 — stratify before concluding anything
+
+An aggregate that looks fine is the normal way these problems hide. Slice every metric by
+**domain, locale, region, surface, section, experiment branch, and client/addon version**, and look
+for a single stratum at zero or down sharply against its own trailing median rather than against
+yesterday. When the report says "some users" with no stratum attached, treat experiment branch,
+region, and rollout state as the first cuts — some surfaces are reachable only through experiment
+enrolment, so an enrolled/unenrolled split is invisible to a locale slice.
+
+Strata are not interchangeable, and the volumes are strongly seasonal; both traps, and the
+per-surface source mix, are detailed in `references/data-sources.md`. Compare each stratum against
+its own history.
+
+**Liveness is not health.** "The job ran successfully" is compatible with total data loss in every
+one of these pipelines. Count what came out, compare it against what went in, and never accept a
+green run as evidence.
+
+If the measurements exonerate the backend, stop at the service boundary. Read far enough to name the
+owning team and the contract that is being broken, then hand it over rather than continuing into a
+system this skill does not cover. A second, unrelated anomaly you trip over on the way is a one-line
+note plus a task, not a second investigation.
+
+## Step 6 — try to break your own conclusion
+
+Before writing anything down as fact, attack it:
+
+- Check the boring explanations first: timezone; a filter in your own query; a `LIMIT` silently
+  truncating; an unrepresentative code path; sampling; a column that does not mean what its name
+  says; a partially-launched feature.
+- Then reach for the falsification moves in `references/failure-modes.md` — the code path, exact
+  enumeration, repetition, the real limit, the config's history — picking the ones that would break
+  your specific mechanism.
+- **Measure the effect, not just the mechanism.** A guard that provably runs is not evidence that
+  the outcome is correct; verify the outcome separately.
+- Cross-check against a second, independent source. One source is a hypothesis.
+- Re-read the step-2 prior. If your conclusion implies that something the prior says is working is
+  broken, re-check your own measurement first — a filter in your query, a stratum mismatch, or a
+  wrong surface identifier is the likelier explanation. If the measurement survives that re-check,
+  the measurement wins: state the contradiction explicitly in FINDINGS.md, note it to the developer
+  in one line, and keep going.
+
+Mark every claim **verified**, **inferred**, or **refuted**. Keep the refuted ones in the document;
+they stop the next person re-running them.
+
+## Step 7 — write FINDINGS.md
+
+```markdown
+# <symptom> — investigation
+
+**Status:** investigating | root cause identified | blocked
+**Class:** outage | degradation | data-quality defect | user-facing bug | near-miss | not-incident
+**Severity:** critical | high | medium | low
+**Affected:** Firefox clients | editors | both — with a magnitude, not an adjective
+
+## Summary
+Three sentences: what broke, since when, what the user-visible effect is.
+
+## Timeline (UTC)
+| Time | Observation | Source |
+
+## Prior
+**Derived** — per-surface volumes, recent deploys and model artifacts, trailing profile, prior
+occurrences. Each with the query or link.
+**Reported** — verbatim answers from the developer, or `asked <time UTC>, no answer received`.
+
+## Evidence
+Numbered findings. Each: the claim, the query or command, the result, and
+verified/inferred/refuted. Reference saved files by path — every number reproducible from this
+directory.
+
+## Root cause
+The mechanism, with the code path or config that produces it. "Not established" when it is not.
+
+## Hypotheses considered and dropped
+Every line you enumerated, and what ruled each one out — or why it was not pursued.
+
+## Impact quantified
+Rows, requests, users, hours, locales. A number, or an explicit "unquantified because X".
+
+## Proposed fix
+The change as a diff in this document, at file-and-line where you can get there, produced by
+reading the clones.
+
+## Detection gap
+The check that would have caught this: the metric or invariant, its threshold shape, where it
+would run, and why it does not already exist.
+
+## Could not measure
+Any telemetry plane you could not reach, and the access that would unblock it.
+```
+
+## Step 8 — close the loop
+
+- **Name the detection gap.** Time-to-detect dominates time-to-diagnose in this system; a check
+  that fires is worth more than a faster diagnosis. `references/failure-modes.md` lists the
+  invariants worth asserting — propose the one that fits, and say where it should live.
+- **Report what you could not reach.** Missing dashboard, IAM, token, or VPN access is a finding,
+  not an inconvenience to route around. Leave any unresolved access task on the list rather than
+  clearing it.
+- **Link prior art** — older investigation, ticket, postmortem, or a related error already known.
+- **Leave the follow-up as a task**, owned by whoever will act: confirm the fix shipped and the
+  metric actually recovered. State that you have done so rather than asking whether to.
+- Hand back a two-line verdict plus the FINDINGS.md path. Do not paste the document into chat.
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
new file mode 100644
index 0000000..e4da4a9
--- /dev/null
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
@@ -0,0 +1,262 @@
+# New Tab (HNT) backend — where to look
+
+Reference for the `hnt-backend-investigation` skill.
+
+## Contents
+
+- The pipeline, and which repo owns each stage
+- Vocabulary: surface, locale, section, and the id to trace items by
+- Sentry — projects and the traps in reading them
+- Merino — live requests and GCP projects
+- BigQuery — the tables worth knowing and the traps in reading them
+- Curated corpus MySQL — access and schema shape
+- The editor plane — admin-api and curated-corpus-api
+- AWS — profiles, and what logs give you that Sentry cannot
+- Zyte — extraction API and the vendor's own stats
+- **Access requests** — the one-line ask for each gated source
+
+Write your own queries: this file gives you the shape of the data and the traps, not canned SQL.
+Access varies by developer. Verify a source is reachable before building a plan around it, and if it
+is not, follow "Don't stall on the developer" in SKILL.md — raise it once as a task and keep
+investigating, rather than stopping or silently substituting a weaker source.
+
+## The pipeline
+
+**crawl / discovery → hydration (Zyte) → curated corpus → ML section assembly → curated-corpus-api →
+client-api → Merino → Firefox New Tab.** Editors act on the corpus through curation-admin-tools →
+admin-api → curated-corpus-api. Telemetry lands in BigQuery.
+
+`client-api` is the Apollo federated router (`Pocket/pocket-monorepo`, `servers/client-api`) and it
+is easy to miss: it has **no Sentry project of its own in either organisation**, so a failure there
+surfaces as a Merino symptom. Merino also reaches it at the **prod** endpoint unconditionally —
+`CorpusApiGraphConfig.endpoint` returns `CORPUS_API_PROD_ENDPOINT` regardless of Merino's own
+environment, and the dev constant beside it is unused — so stage Merino reads prod corpus data.
+
+| Repo | GitHub | Role |
+|---|---|---|
+| `merino-py` | mozilla-services/merino-py | Serves New Tab recommendations and Firefox Suggest |
+| `content-monorepo` | Pocket/content-monorepo | Curated corpus, recommendations, section manager |
+| `content-ml-services` | mozilla/content-ml-services | Crawl, classification, section assembly (Metaflow, Cloud Functions) |
+| `pocket-monorepo` | Pocket/pocket-monorepo | client-api federated router, shared infrastructure |
+| `curation-admin-tools` | Pocket/curation-admin-tools | Editor-facing web app |
+| `admin-api` | Pocket/admin-api | Federated GraphQL gateway for the admin tools |
+| `bigquery-etl` | mozilla/bigquery-etl | New Tab engagement and Merino recommendation ETL |
+| `serverless-image-cache` | Pocket/serverless-image-cache | Thumbor image resize and cache |
+| `firefox` | mozilla-firefox/firefox | Client side of the contract (`browser/extensions/newtab`) |
+
+Locate clones rather than assuming paths:
+`find ~ -maxdepth 3 -type d -name .git -print0 2>/dev/null | xargs -0 -n1 dirname`. For a repo that
+is not cloned, read it through `gh api` or `gh search code`.
+
+## Vocabulary: surface, locale, section
+
+A **surface** is one locale/market feed of the corpus, written `NEW_TAB_EN_US`. A **section** is a
+topic row inside a surface. Merino's request `locale` is hyphenated (`en-US`) and the surface is
+**derived** from language plus region — the mapping table is `SurfaceId` in
+`merino/curated_recommendations/localization.py`, not a reformatting of the locale string. Some
+surfaces are reachable only through experiment enrolment. Stratifying by locale and stratifying by
+surface are therefore not the same slice.
+
+Merino appends `utm_source=firefox-newtab-<surface>` to item URLs (`get_utm_source` and
+`update_url_utm_source` in `curated_recommendations/corpus_backends/utils.py`), so a URL taken from a
+response will never match a stored `url`. Trace items by the stable id instead: the response's
+`corpusItemId` is the corpus GraphQL item `id`, which lines up with `ApprovedItem.externalId` in
+corpus MySQL and `approved_corpus_item_external_id` in BigQuery. Confirm that last hop on the first
+item you trace rather than assuming it.
+
+## Sentry
+
+Reachable through the Sentry MCP server if one is connected (its tools are prefixed `mcp__sentry__`).
+Prefer the event-search tool over issue-search when you need a volume breakdown by error message —
+issue search alone will not decompose an umbrella fingerprint. If no Sentry tools are present, treat
+it as a blocked source.
+
+The HNT services are in the **`mozilla`** organisation, prefixed `hnt-`; issue short-ids look like
+`HNT-CRAWL-9`.
+
+`hnt-crawl` · `hnt-metaflow` · `hnt-curated-corpus-api` · `hnt-admin-api` ·
+`hnt-curation-admin-tools` · `hnt-section-manager-lambda` · `hnt-corpus-scheduler-lambda` ·
+`hnt-prospect-api` · `hnt-prospect-translation-lambda` · `hnt-serverless-image-cache` ·
+`hnt-braze-content-proxy` · `hnt-feature-flags`
+
+Merino is separate: project `merino-py`, also in `mozilla`. Some older projects exist in the `pocket`
+org; prefer the `hnt-` ones for anything current. Enumerate the current projects before relying on
+this list — a project that does not resolve is a naming change, not evidence of zero errors.
+
+Traps:
+
+- **`first seen` is bounded by retention**, so it is not onset. Nearly every long-running issue
+  reports a first-seen date at the edge of the retention window.
+- Issue volume is dominated by long-standing error floors. Establish what was already there before
+  attributing anything to today, and check whether a floor *changed* rather than whether it exists.
+- Decompose an issue by error message before trusting its title or trending it.
+- Absence of events is weak evidence: a process that dies at startup, or a job that drops items
+  while reporting success, emits nothing.
+
+## Merino
+
+Reproducing the client call is often the fastest confirmation of a client-visible symptom:
+`POST https://merino.services.mozilla.com/api/v1/curated-recommendations` with a JSON body carrying
+`locale`, `region`, `topics`, `sections`, `feeds` (e.g. `["sections"]`), `enableInterestPicker`, and
+optionally `experimentName` / `experimentBranch` to land on an experiment branch. Send a realistic
+Firefox `User-Agent`. Save every response. First things to check on the payload: section count,
+items per section, presence of `followable` / `allowAds`, and the age of the newest item.
+
+Keep it to the requests you need. If you find yourself writing a loop over locales or branches
+against prod, query the telemetry instead. Stage answers payload-shape questions — but remember it
+reads prod corpus data, so it cannot answer questions about prod corpus content.
+
+GCP projects, for log reads, metrics, and GCS listings — confirm against
+`merino/configs/production.toml` **and `merino/configs/stage.toml`** rather than trusting this table:
+
+| Environment | Project |
+|---|---|
+| Merino prod | `moz-fx-merino-prod-5de4`; some buckets in `moz-fx-merino-prod-1c2f` |
+| Merino stage | `moz-fx-merino-nonprod-ee93`; also `moz-fx-merino-nonprod-db57` |
+
+When the logging API returns nothing useful, the BigQuery log sink for the same project usually does.
+That sink table is far larger than anything in the BigQuery list below — check whether it has a
+partition column and filter on it, aggregate inside the query rather than pulling rows, and
+`--dry_run` first.
+
+## BigQuery
+
+Confirm the billing project with a `--dry_run` before the first real query; a personal sandbox is
+usually `moz-fx-dev-<ldap>-sandbox`. If you cannot confirm one, raise it as an access task.
+
+| Table | What it is for |
+|---|---|
+| `moz-fx-mozsoc-ml-prod.prod_rss_news.rss_feed_items` | Articles discovered by crawl — per-surface, per-source (`PAGE` vs `RSS`) volume over time |
+| `moz-fx-mozsoc-ml-prod.prod_articles.zyte_cache` | Hydrated article metadata keyed on `canonical_url` — what the pipeline believes an article says |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.sections_v1` | Section existence, enable/disable state, and surface, as an event stream |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.section_items_v1` | Which items sit in which section, and when each was last touched — the freshness check for a stalled section |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.corpus_items_current_v1` | Current-state view of corpus items — prefer this to reducing the event table yourself |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.approved_corpus_items` / `scheduled_corpus_items` | Maintained current-state views for approved and scheduled items |
+
+Confirm each table still exists before drawing a conclusion from an empty result.
+
+Traps that will silently give you a wrong answer:
+
+- **There is no domain column on `rss_feed_items`.** Derive the domain from `canonical_url` (or
+  `origin_url` for the page it was found on). A per-domain funnel is a computed grouping, not a
+  lookup.
+- **`source` is null for everything crawled before 2025-08-11** — 6.8M of 22.6M rows. A long
+  per-source series will show `PAGE` and `RSS` springing into existence on that date; that is the
+  column being introduced, not the crawl changing.
+- **`crawled_date` and `published_date` are STRING; `crawled_at`, `published_at` and `loaded_at` are
+  TIMESTAMP.** Use the timestamps for any time arithmetic.
+- **Cost does not work the way you expect.** `rss_feed_items` (~23.6 GB) and `zyte_cache` (~128 GB)
+  are **unpartitioned and unclustered**, so a date predicate reduces nothing — only column selection
+  does. Never `SELECT *` on them and always `--dry_run` first. The `snowflake_migration_derived`
+  tables are day-partitioned on `happened_at` but do not require a partition filter, so supply one
+  yourself.
+- **The `*_v1` event tables are event logs, not current state.** Rows accumulate per change, so a
+  plain `COUNT(*)` over-counts. Prefer the current-state views above; where you must reduce the log
+  yourself, take the latest row per id and check the inflation ratio rather than assuming it.
+- **`sections_v1.source` is unusable for current sections** — null on 81% of all sections and on
+  **every** currently-active one. Section ownership (`MANUAL` vs `ML`) comes from
+  `Section.createSource` / `updateSource` / `deactivateSource` in corpus MySQL. A null `source` means
+  unknown, never `MANUAL`.
+- **Surface identifiers differ by table.** Crawl data uses `en_US`; section data uses
+  `NEW_TAB_EN_US`. Joining or comparing them naively produces empty results that look like an outage.
+- **The crawl surface set and the serving surface set are not identical**, so a crawled surface with
+  no sections is not automatically a bug.
+- **Source mix varies by surface** — some have no RSS-sourced content, some no page-crawled content.
+  Judge each surface against its own history.
+- **Volume is strongly day-of-week seasonal.** Compare against a trailing median, never against
+  yesterday.
+- A zero in the crawl table is ambiguous between "found nothing" and "every request failed"; there is
+  no attempt/error record to disambiguate it.
+
+## Curated corpus MySQL
+
+Production database behind curated-corpus-api, reached through a preconfigured read-only login path.
+Check what exists with `mysql_config_editor print --all`, and expect to need VPN — a hang rather than
+an auth error is the usual symptom of being off it.
+
+Useful invocation guards: `--safe-updates` (caps returned rows at 1000 and aborts queries estimated
+to examine over a million) and `SET SESSION max_execution_time=10000` (10s server-side cap). The
+1000-row cap **truncates silently**, so add an explicit `LIMIT` or raise the cap when you need a
+complete set.
+
+Schema `curation_corpus`. The tables that matter: `ApprovedItem` (the corpus itself, keyed by
+`externalId` and `url`), `SectionItem` and `Section` (placement and section config — `Section` also
+carries `createSource` / `updateSource` / `deactivateSource`, the authoritative ML-vs-manual
+ownership), `ScheduledItem` (surface scheduling), `RejectedCuratedCorpusItem`, and the
+`PublisherDomain` / `TrustedDomain` / `ExcludedDomain` domain lists. `SectionItem` runs into the
+millions of rows and `ApprovedItem` into the hundreds of thousands — check indexes with `SHOW INDEX`
+before filtering, since several obvious filter columns are unindexed.
+
+## The editor plane
+
+When the symptom is editor-facing, reproducing it means an authenticated editor session, which is an
+interactive login and therefore the developer's errand rather than yours. What you can do alone:
+`hnt-admin-api` and `hnt-curated-corpus-api` in Sentry for the failure window, the corpus rows the
+mutation would have touched, and the resolver itself in `Pocket/admin-api` and
+`Pocket/content-monorepo`. That is usually enough to name the mechanism without a session.
+
+## AWS
+
+content-monorepo infrastructure runs in AWS. Discover profiles with
+`grep -E '^\[profile' ~/.aws/config` and confirm one works with `sts get-caller-identity`; match prod
+against dev to the environment you are investigating.
+
+CloudWatch Logs Insights is frequently the source that cracks a case — it will give you total
+operation counts and per-error-type breakdowns that Sentry structurally cannot, because Sentry only
+sees what was raised. It bills per GB scanned per query, so pass the narrowest
+`--start-time`/`--end-time` that could answer the question and `stats`-aggregate rather than dumping
+`fields`. Also useful: alarm history (transition timestamps and state-reason margins), and pulling an
+anomaly band itself as a metric-math series to compare its predicted centre against reality.
+
+SSO sessions expire and the login is interactive, so it has to be the developer: see the table below.
+
+## Zyte
+
+Two separate APIs, both **metered — every call costs money**. Save all responses; never bulk-crawl to
+satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`, and reference it as
+`$ZYTE_API_KEY` in anything you save, so no key is written into a query file or the transcript.
+
+**Extraction API** (`ZYTE_API_KEY`) reproduces what the crawler saw for a URL. Request `article` for
+a single page or `articleList` for an index page — not both — and put `extractFrom:
+"httpResponseBody"` inside `articleOptions` / `articleListOptions` to skip the browser. Check
+**`statusCode` and the extraction probability before anything else**: a non-2xx, a bot wall, or a
+"JavaScript is disabled" page still returns a populated object that reads like success. Probability
+lives at `article.metadata.probability` for a single page and per item at
+`articleList.articles[].metadata.probability` — `articleList.metadata` carries only
+`dateDownloaded`. Compare `canonicalUrl` against the URL you requested; cross-domain canonicals pull
+unapproved domains into the corpus.
+
+**Stats API** (`ZYTE_SECRET_KEY`, `https://zyte-api-stats.zyte.com/api/stats`) is the vendor's own
+view of your traffic: per-domain response-code distribution over time. `organization_id` is
+**required** and auth is HTTP basic with the secret key as the username and an empty password, so a
+bare GET on the endpoint fails validation — take the org id from the production caller
+(`scripts/fetch/zyte_stats.py` in `content-ml-services`) rather than guessing. Only `groupby_time`
+and `groupby_domain` group; `response_codes` is a *filter*, not a grouping, and
+`include_domain_health=true` is rejected without `groupby_domain=true`. This is the right source for
+"did this domain start failing, and when" — your own logs will not show it if the pipeline discards
+non-allowlisted status codes.
+
+Freshness thresholds are a common thing to want here and a common thing to invent. Derive the
+intended cadence from the assembly schedule in `content-ml-services` and the corpus-scheduler-lambda
+trigger config rather than guessing, and record the interval you used in FINDINGS.md.
+
+## Access requests
+
+Raise one of these only when that source is one of your strongest current lines, then keep working.
+Each is a short errand: give the developer the command, not a description of the problem.
+
+| Blocked source | Ask them to |
+|---|---|
+| Corpus MySQL hangs rather than erroring | Connect to VPN. If it still hangs, ask whether the read-only login path is still valid |
+| No read-only MySQL login path configured | Set one up, or give you the host and read-only user to configure |
+| AWS SSO session expired | Run `! aws --profile <profile> sso login` in-session, so the output lands here |
+| No AWS profile at all | Say which read-only profile they have, or request one for the account you need |
+| BigQuery permission denied on a dataset | Confirm which billing project to use, or request read access to the dataset |
+| `gcloud` lacks access to a Merino project | Request viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer |
+| Zyte key missing from the environment | Export it in the shell they launched from and restart the session, or run the one extraction you need and paste back the JSON — the JSON, not the key |
+| Sentry MCP unavailable or scoped too narrowly | Authenticate the Sentry MCP server for the `mozilla` org, or paste the issue's event counts broken down by error message |
+| Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time |
+| The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for |
+
+Notice the last row: when a dashboard, explore, or console holds the answer, asking for *access* is
+usually the slower path. Asking a precise question about what it shows is faster for both of you.
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
new file mode 100644
index 0000000..83e1e15
--- /dev/null
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
@@ -0,0 +1,146 @@
+# How New Tab (HNT) backend failures behave
+
+Reference for the `hnt-backend-investigation` skill: the shapes these failures take, the invariants
+worth asserting, and the moves that kill a wrong hypothesis. Read this before you take any
+hypothesis seriously.
+
+## Contents
+
+- Classify the shape first — failure classes and what each looks like on first contact
+- If it came from an alert, audit the alert
+- Why these failures are silent — the mechanisms that turn a failure into a valid-looking result
+- Falsification moves that work — what to reach for when a story feels too clean
+- Structural gaps to report rather than infer past
+- Invariants worth asserting — hard checks, and the class each one detects
+
+## Classify the shape first
+
+The tells below are heuristics for orienting, not proof. Use them to widen the hypothesis list at
+step 3, not to shortcut it.
+
+| Failure class | What you tend to see first |
+|---|---|
+| Vendor extraction-quality defect | Status codes fine and error rate flat, but a content-shape ratio per domain drifts — empty-body share, title length, headline churn |
+| Upstream unavailability | Your own logs contain *nothing*; a per-domain count goes to zero with no errors logged. Visible only in the vendor's own status data |
+| Silent validation rejection | `items_written < items_received`, exit status success, no error raised |
+| Resource exhaustion / capacity cliff | Output stops abruptly and completely, tracking input size or scale rather than time of day |
+| ML model or routing failure | Job succeeded and the aggregate gate passed, while one class's output is zero and its siblings are current |
+| Data-model / identity defect | Counts look normal but the rows are wrong — the same content living under two keys (URL, canonical, slug, external id) |
+| Filter / threshold misconfiguration | The affected population clusters exactly on a boundary value; or a global aggregate breaches while every stratum is fine |
+| Unscoped bulk database operation | An entire entity's row count changes at one timestamp, with nothing thrown |
+| Cascading failure / retry amplification | Outbound attempts per key far above the design rate; error volume scales with fleet size rather than user traffic |
+| Observability defect | The alert's own numbers disagree with an independent measurement of the same thing; the alarm fires while service metrics are healthy |
+| Analytics correctness defect | A derived dimension disagrees with the entity it claims to describe |
+| Dedup failure | Distinct identifiers, identical assets — hash the bytes, not the URL |
+| Authorization defect | Symptom is both user-specific and surface-specific; peers on the same surface are unaffected |
+| Client / product defect | Backend telemetry is normal and the symptom is banded by client version or build |
+| Capacity near-miss | Nothing is broken; the only signal is consumption against a documented quota |
+
+The Observability defect row is expanded in the next section.
+
+A calibration note: symptoms that originate from a monitor's output or from a prevailing worry are
+the ones that most often turn out not to be real, while content a human actually observed usually is.
+This is a reason to confirm the signal independently in both directions, not a reason to dismiss
+either.
+
+## If it came from an alert, audit the alert
+
+Alert quality is its own failure class, and worth a few minutes before chasing the system. This is an
+addition to confirming the symptom independently, not a substitute for it.
+
+- **Wrong layer** — the alert watches a gateway while the loss happens a hop upstream.
+- **Umbrella fingerprint** — one issue collapses several unrelated failures; decompose by error
+  message before trusting the title or the trend, and anchor on stable issue identifiers rather than
+  numeric ids in the alert body.
+- **Mis-centred or flapping band** — repeated transitions at a fixed offset within the hour, and
+  margins under a percent, mean the band, not the system.
+- **Counting artefact** — a panel that counts per open period rather than per calendar day can be off
+  by an order of magnitude.
+- **Unroutable** — if you cannot tell prod from staging at a glance, fix that first.
+
+## Why these failures are silent
+
+Not under-monitored — structurally unobservable. The recurring mechanism is that an upstream failure
+gets converted into a valid-looking empty or partial result. Expect these:
+
+- A pipeline that keeps only an allowlist of status codes and swallows request errors, so a total
+  block is indistinguishable from "nothing found today".
+- A vendor non-2xx arriving as a partially-filled success object, so a total outage raises nothing.
+- A job reporting invocation-level success while silently dropping every item it received.
+- Input-validation rejections that are logged and never surfaced.
+- An `INNER JOIN` against a frozen upstream catalog shedding rows quietly instead of failing.
+- A process that dies during startup, which emits no error event at all.
+- A bulk write with no row-count guard.
+- An aggregate quality gate passing while one class collapses beneath it.
+- An alarm that treats missing data as "missing" rather than as a breach, sitting against an emitter
+  that publishes nothing at zero — so it goes *quiet* during a total stop.
+
+The consequence for every one of these: **liveness is not health.** Count what came out and compare
+it to what went in.
+
+## Falsification moves that work
+
+Each of these has killed a plausible, confidently-held hypothesis. Reach for them when a story feels
+too clean.
+
+- **Read the code path.** Assumed behaviour of a resolver, filter, or job is the single most common
+  source of a wrong mechanism.
+- **Enumerate the whole population** rather than sampling, when it is free to count — in BigQuery or
+  MySQL, "0 of N" is a different claim from "none in my sample". Against a metered API, sample, say
+  what the sample is, and put any larger enumeration on the task list with its call count.
+- **Repeat the observation N times** before calling anything non-deterministic, then vary one input
+  at a time.
+- **Read the actual limit** — quota, timeout, payload cap, row cap — instead of assuming which one
+  binds. Adjacent limits on the same resource are easy to confuse.
+- **Check the history of the constant or config** you are relying on; a value that used to be true is
+  the classic stale-assumption trap.
+- **Dump the attribute across the entire population** when you suspect truncation or a per-record
+  limit.
+- **Go to the other side's telemetry.** A vendor's or upstream's own status data will show what your
+  logs structurally cannot.
+- **Compare against sibling strata** at the same layer — the healthy peers localise the fault faster
+  than reading code does.
+- **Design a canary that would falsify** the hypothesis rather than a test that would confirm it.
+- **Refute it yourself, in writing.** Before a conclusion leaves FINDINGS.md, draft the strongest
+  case against it under "Hypotheses considered and dropped" and answer that case with a measurement.
+  A second agent handed the evidence and told to break the conclusion is cheaper than waiting for a
+  human reviewer.
+
+## Structural gaps to report rather than infer past
+
+These make certain questions unanswerable from stored data alone. When one blocks you, say so.
+
+- No crawl attempt/success/error record, so a zero row count is ambiguous between "found nothing" and
+  "every request failed".
+- No lifecycle metadata or tombstones on the crawl target list, so an intentional retirement is
+  indistinguishable from an outage.
+- No persisted ML filter funnel, so per-stage drop-off cannot be reconstructed afterwards.
+- No queryable deployment history to line a timeline up against.
+- No status code or extraction-probability stored alongside cached hydration results, so a stored row
+  does not imply a successful extraction.
+
+## Invariants worth asserting
+
+Hard equalities and floors rather than statistical thresholds, so they fire with near-zero false
+positives. When you finish an investigation, propose the one that fits — or add a new one shaped like
+these.
+
+| Invariant | Class it detects |
+|---|---|
+| Articles/hour > 0 per surface **per source**, judged against that surface's own mix | Upstream unavailability; capacity cliff |
+| Active items per fixed-size section == configured N | Silent validation rejection; model failure |
+| `items_written == items_received` per job invocation | Silent validation rejection |
+| Input-validation reject rate == 0 | Silent validation rejection |
+| Newest active item age per section within that surface's assembly interval | Model or routing failure; capacity cliff |
+| Vendor non-2xx share per domain below a floor | Upstream unavailability; extraction defect |
+| Content-shape ratio per domain within bounds (empty-body %, title length) | Extraction-quality defect |
+| Per-domain funnel positive at every stage: discovered → hydrated → approved | Upstream unavailability; filter misconfiguration |
+| Canonical domain ∈ approved list | Data-model / identity defect |
+| Per-class model recall above a floor before publish | Model or routing failure |
+| Row-count delta on key tables within bounds | Unscoped bulk operation |
+| Every bound parameter in shared SQL supplied by every caller | Analytics correctness |
+| Consumption below documented quota, alerting on the ratio | Capacity near-miss |
+| The alarm itself emits a value at zero | Observability defect |
+
+Stratify anything you assert. A rule that holds globally can be false for one surface, because the
+surfaces do not all draw content the same way.

From d0e16f9d3f8d1e64ad454f159be6929efa328313 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 13:47:28 -0400
Subject: [PATCH 02/10] newtab: shorten hnt-backend-investigation README entry

---
 plugins/newtab/README.md | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md
index e13bd86..80e43ea 100644
--- a/plugins/newtab/README.md
+++ b/plugins/newtab/README.md
@@ -12,8 +12,17 @@ Use this skill when adding cleanup comments to Nova-related changes. See [`skill
 
 ### `hnt-backend-investigation`
 
-Diagnoses Home New Tab **backend** errors, outages, and data-quality defects — Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline.
+Diagnostic workflow for Home New Tab **backend** incidents, covering Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Its output is an evidence-backed `FINDINGS.md` carrying a root cause, quantified impact, and the check that would have caught the problem earlier.
 
-Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. It confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, and writes an evidence-backed `FINDINGS.md` naming the check that would have caught the problem earlier. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details.
+Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details.
 
-**Needs:** `gh`; `gcloud`/`bq` with a billing project; the `aws` CLI with a read-only SSO profile; a MySQL client plus Mozilla VPN; `uv` or `python3`; the Sentry MCP server if you have it. The skill runs on whatever subset you have — anything unreachable is raised as a task and reported under "Could not measure".
+**Optional tools:**
+
+- `gh`
+- `gcloud` and `bq`
+- the `aws` CLI, with a read-only SSO profile
+- a MySQL client, plus Mozilla VPN
+- `uv` or `python3`
+- the Sentry MCP server
+
+Anything unreachable is raised as a task and reported under "Could not measure".

From 64b29b4b7f09c9716cc4e419706b327faeb71546 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 13:54:21 -0400
Subject: [PATCH 03/10] newtab/hnt-backend-investigation: drop detection-gap
 scope

---
 plugins/newtab/README.md                         |  2 +-
 .../skills/hnt-backend-investigation/SKILL.md    | 16 ++++------------
 .../references/failure-modes.md                  | 16 ++++++++--------
 3 files changed, 13 insertions(+), 21 deletions(-)

diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md
index 80e43ea..9674925 100644
--- a/plugins/newtab/README.md
+++ b/plugins/newtab/README.md
@@ -12,7 +12,7 @@ Use this skill when adding cleanup comments to Nova-related changes. See [`skill
 
 ### `hnt-backend-investigation`
 
-Diagnostic workflow for Home New Tab **backend** incidents, covering Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Its output is an evidence-backed `FINDINGS.md` carrying a root cause, quantified impact, and the check that would have caught the problem earlier.
+Diagnostic workflow for Home New Tab **backend** incidents, covering Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Its output is an evidence-backed `FINDINGS.md` carrying a root cause and quantified impact.
 
 Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details.
 
diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
index a4437b7..bb24b95 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -1,6 +1,6 @@
 ---
 name: hnt-backend-investigation
-description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc naming the check that would have caught it. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale.
+description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale.
 ---
 
 # New Tab (HNT) Backend Investigation
@@ -9,15 +9,14 @@ Diagnose a problem in the services behind Firefox Home New Tab recommendations 
 curated corpus, admin-api, the article crawler, the ML section pipeline, the data pipelines. Not
 for in-tree `browser/extensions/newtab` frontend bugs.
 
-Your product is a **diagnosis**: a root cause, the evidence for it, the impact quantified, and the
-check that would have caught it earlier. Two populations can be affected, and every finding should
-say which: **Firefox New Tab clients** (everyone who sees recommendations) and **editors/curators**
+Your product is a **diagnosis**: a root cause, the evidence for it, and the impact quantified. Two
+populations can be affected, and every finding should say which: **Firefox New Tab clients** (everyone who sees recommendations) and **editors/curators**
 (the internal editorial team working through curation-admin-tools and admin-api).
 
 **How to reach each system, and the traps that will silently give you wrong answers** — read before
 your first probe in step 1: [references/data-sources.md](references/data-sources.md)
 
-**How these systems fail, the invariants worth asserting, and falsification moves that work** — read
+**How these systems fail, the invariants to check, and falsification moves that work** — read
 before you enumerate hypotheses in step 3, because it will change which ones you take seriously:
 [references/failure-modes.md](references/failure-modes.md)
 
@@ -290,19 +289,12 @@ Rows, requests, users, hours, locales. A number, or an explicit "unquantified be
 The change as a diff in this document, at file-and-line where you can get there, produced by
 reading the clones.
 
-## Detection gap
-The check that would have caught this: the metric or invariant, its threshold shape, where it
-would run, and why it does not already exist.
-
 ## Could not measure
 Any telemetry plane you could not reach, and the access that would unblock it.
 ```
 
 ## Step 8 — close the loop
 
-- **Name the detection gap.** Time-to-detect dominates time-to-diagnose in this system; a check
-  that fires is worth more than a faster diagnosis. `references/failure-modes.md` lists the
-  invariants worth asserting — propose the one that fits, and say where it should live.
 - **Report what you could not reach.** Missing dashboard, IAM, token, or VPN access is a finding,
   not an inconvenience to route around. Leave any unresolved access task on the list rather than
   clearing it.
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
index 83e1e15..8b6765b 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
@@ -1,8 +1,8 @@
 # How New Tab (HNT) backend failures behave
 
 Reference for the `hnt-backend-investigation` skill: the shapes these failures take, the invariants
-worth asserting, and the moves that kill a wrong hypothesis. Read this before you take any
-hypothesis seriously.
+to check, and the moves that kill a wrong hypothesis. Read this before you take any hypothesis
+seriously.
 
 ## Contents
 
@@ -11,7 +11,7 @@ hypothesis seriously.
 - Why these failures are silent — the mechanisms that turn a failure into a valid-looking result
 - Falsification moves that work — what to reach for when a story feels too clean
 - Structural gaps to report rather than infer past
-- Invariants worth asserting — hard checks, and the class each one detects
+- Invariants to check — hard checks, and the class each one points to
 
 ## Classify the shape first
 
@@ -119,13 +119,13 @@ These make certain questions unanswerable from stored data alone. When one block
 - No status code or extraction-probability stored alongside cached hydration results, so a stored row
   does not imply a successful extraction.
 
-## Invariants worth asserting
+## Invariants to check
 
-Hard equalities and floors rather than statistical thresholds, so they fire with near-zero false
-positives. When you finish an investigation, propose the one that fits — or add a new one shaped like
-these.
+Hard equalities and floors rather than statistical thresholds, so a violation is unambiguous rather
+than a matter of degree. Assert the ones that bear on your hypotheses against the data: a broken one
+localises the fault immediately, and a holding one eliminates a line cheaply.
 
-| Invariant | Class it detects |
+| Invariant | Class it points to |
 |---|---|
 | Articles/hour > 0 per surface **per source**, judged against that surface's own mix | Upstream unavailability; capacity cliff |
 | Active items per fixed-size section == configured N | Silent validation rejection; model failure |

From ae78f255f13d14ee4162d2e493f14d20ded5dd75 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 14:05:19 -0400
Subject: [PATCH 04/10] newtab/hnt-backend-investigation: derive step-1 facts
 from data

---
 .../skills/hnt-backend-investigation/SKILL.md   | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
index bb24b95..f8c4eca 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -96,10 +96,9 @@ the same symptom. Say which directory you chose; do not ask.
 
 ## Step 1 — pin down the report
 
-Establish these five facts. Derive what you can — the alert or issue carries the symptom and its
-volume, stratifying the data gives you the surface and locale, a live request tells you whether it
-is still happening. Ask only for what is genuinely undiscoverable, in one batch, and keep working
-while you wait.
+Establish these five facts from the data. The alert or issue carries the symptom and its volume,
+stratifying gives you the surface and locale, and a live request tells you whether it is still
+happening.
 
 | Fact | Why it matters |
 |---|---|
@@ -109,8 +108,6 @@ while you wait.
 | Surface, locale, section, client version | The stratum is very often the diagnosis |
 | Still happening right now? | Live reproduction vs. historical forensics |
 
-Then confirm it yourself.
-
 **If it is still happening, capture the perishable evidence first** — a live
 `curated-recommendations` request, current logs, current per-section counts — and save the raw
 responses under `api_responses/`. Build the step-2 prior while those probes are in flight; history
@@ -153,10 +150,10 @@ your own inference. Build that prior yourself before asking for it — most of i
 - **Whether it has happened before** — search Sentry for the same signature, and the service repos'
   GitHub issues.
 
-What is left is genuinely in the developer's head, and it is worth one short, non-blocking ask:
-anything in flight that would not show up in a repo or a dashboard, and whether a postmortem or
-incident register exists for this area. An existing write-up can answer in a paragraph what costs
-hours to reconstruct, so ask early — then carry on without waiting.
+What is left is genuinely in the developer's head: anything in flight that a repo or a dashboard
+would not show, and whether a postmortem or incident register already covers this area. Raise that
+as one short note and carry on without waiting; an existing write-up can answer in a paragraph what
+costs hours to reconstruct.
 
 Record what you derive and what they tell you in FINDINGS.md, keeping the two apart. When a later
 query contradicts the prior, treat your own inference as the suspect first, and say so.

From c5d9d8f04e810b1fa0702b54237aa36952699996 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 14:13:03 -0400
Subject: [PATCH 05/10] newtab/hnt-backend-investigation: derive hypotheses
 from data, drop symptom lookup

---
 .../skills/hnt-backend-investigation/SKILL.md | 51 +++++++++----------
 .../references/failure-modes.md               | 36 +++++--------
 2 files changed, 36 insertions(+), 51 deletions(-)

diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
index f8c4eca..ff9ea09 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -16,9 +16,9 @@ populations can be affected, and every finding should say which: **Firefox New T
 **How to reach each system, and the traps that will silently give you wrong answers** — read before
 your first probe in step 1: [references/data-sources.md](references/data-sources.md)
 
-**How these systems fail, the invariants to check, and falsification moves that work** — read
-before you enumerate hypotheses in step 3, because it will change which ones you take seriously:
-[references/failure-modes.md](references/failure-modes.md)
+**How these systems fail, the invariants to check, and falsification moves that work** — read when a
+probe comes back empty, when a story starts looking too clean, or when your hypothesis list has
+narrowed to one: [references/failure-modes.md](references/failure-modes.md)
 
 ## How to think about this
 
@@ -158,31 +158,26 @@ costs hours to reconstruct.
 Record what you derive and what they tell you in FINDINGS.md, keeping the two apart. When a later
 query contradicts the prior, treat your own inference as the suspect first, and say so.
 
-## Step 3 — enumerate hypotheses, then probe them in parallel
-
-List every plausible explanation before testing any of them. Widen the list from
-`references/failure-modes.md` — its failure-class tells cover classes the symptom table below does
-not, and "Why these failures are silent" names the mechanisms that make a broken pipeline return a
-valid-looking empty result rather than an error.
-
-| Symptom | Likely layer | Probe |
-|---|---|---|
-| Recommendations 500 / timeout / stale | Merino, or its upstream corpus API | Sentry + live `curated-recommendations` request |
-| Section empty or short | ML section assembly → corpus → Merino | Active items per section; freshest item timestamp per section |
-| Section missing entirely | Section config / section manager | Section rows for the surface; section-manager errors |
-| Wrong / stale / duplicate items | Corpus content or ranking | Trace one item end to end by `corpusItemId` |
-| Bad title, image, author, date | Crawl + hydration | Compare the stored item against a live re-extraction |
-| Whole publisher or domain absent | Crawl discovery / domain block | Per-domain counts over time, then the vendor's own status data |
-| Errors up, no reported user impact | The erroring stage itself | Assert that stage's output invariant downstream, then decompose the errors |
-| Editor-facing error or lockout | admin-api → corpus API | Sentry for the editor-facing services; reproduce the mutation |
-| Metrics or dashboard wrong | ETL / derived tables | Compare the derived table against its source |
-
-Before firing, write beside each hypothesis the result that would kill it. Then issue the probes as
-independent tool calls in a single batch, backgrounding anything slow and handing a line that needs
-several dependent steps to a subagent so the batch still returns together. Cap the first wave at
-four or five cheap, independent probes, and hold metered calls and large scans for the second wave.
-Name each `queries/` and `results/` pair after its hypothesis, so a result cannot be attributed to
-the wrong line.
+## Step 3 — form hypotheses from the data, then probe them in parallel
+
+Hypotheses come from what you have already measured, not from a catalogue. Start where step 1 put
+you: the stage whose output is wrong, the stratum that is affected, the moment it changed. Ask what
+could produce exactly that, follow the data one hop upstream, and let each result generate the next
+question. A hypothesis you cannot tie to something you have observed is a guess competing for the
+same probe budget as one you can.
+
+Work the live ones in parallel rather than serially. Before firing, write beside each hypothesis the
+result that would kill it. Then issue the probes as independent tool calls in a single batch,
+backgrounding anything slow and handing a line that needs several dependent steps to a subagent so
+the batch still returns together. Cap the first wave at four or five cheap, independent probes, and
+hold metered calls and large scans for the second wave. Name each `queries/` and `results/` pair
+after its hypothesis, so a result cannot be attributed to the wrong line.
+
+Then look at what came back and do it again. Each round should either kill a line or sharpen the next
+question; when a round kills everything, the data has moved you upstream rather than left you stuck.
+If you are down to a single surviving hypothesis, that is the moment to check the failure classes and
+silent-failure mechanisms in `references/failure-modes.md` against what you have measured, as a guard
+against having narrowed too early.
 
 Two moves consistently break cases open: **leave the tool you started in** — when a source stops
 yielding, measure the same thing somewhere else, since the failure is often invisible in the plane
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
index 8b6765b..d0e8eb3 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
@@ -6,37 +6,27 @@ seriously.
 
 ## Contents
 
-- Classify the shape first — failure classes and what each looks like on first contact
+- Failure classes — the vocabulary, for checking you have not narrowed too early
 - If it came from an alert, audit the alert
 - Why these failures are silent — the mechanisms that turn a failure into a valid-looking result
 - Falsification moves that work — what to reach for when a story feels too clean
 - Structural gaps to report rather than infer past
 - Invariants to check — hard checks, and the class each one points to
 
-## Classify the shape first
+## Failure classes
 
-The tells below are heuristics for orienting, not proof. Use them to widen the hypothesis list at
-step 3, not to shortcut it.
+Vocabulary, not a lookup. Do not use this list to guess a cause from a symptom; derive the layer from
+what you have measured. It is here so that when your hypothesis list has narrowed to one or two, you
+can check it against the range of things that actually break in this stack:
 
-| Failure class | What you tend to see first |
-|---|---|
-| Vendor extraction-quality defect | Status codes fine and error rate flat, but a content-shape ratio per domain drifts — empty-body share, title length, headline churn |
-| Upstream unavailability | Your own logs contain *nothing*; a per-domain count goes to zero with no errors logged. Visible only in the vendor's own status data |
-| Silent validation rejection | `items_written < items_received`, exit status success, no error raised |
-| Resource exhaustion / capacity cliff | Output stops abruptly and completely, tracking input size or scale rather than time of day |
-| ML model or routing failure | Job succeeded and the aggregate gate passed, while one class's output is zero and its siblings are current |
-| Data-model / identity defect | Counts look normal but the rows are wrong — the same content living under two keys (URL, canonical, slug, external id) |
-| Filter / threshold misconfiguration | The affected population clusters exactly on a boundary value; or a global aggregate breaches while every stratum is fine |
-| Unscoped bulk database operation | An entire entity's row count changes at one timestamp, with nothing thrown |
-| Cascading failure / retry amplification | Outbound attempts per key far above the design rate; error volume scales with fleet size rather than user traffic |
-| Observability defect | The alert's own numbers disagree with an independent measurement of the same thing; the alarm fires while service metrics are healthy |
-| Analytics correctness defect | A derived dimension disagrees with the entity it claims to describe |
-| Dedup failure | Distinct identifiers, identical assets — hash the bytes, not the URL |
-| Authorization defect | Symptom is both user-specific and surface-specific; peers on the same surface are unaffected |
-| Client / product defect | Backend telemetry is normal and the symptom is banded by client version or build |
-| Capacity near-miss | Nothing is broken; the only signal is consumption against a documented quota |
-
-The Observability defect row is expanded in the next section.
+vendor extraction-quality defect · upstream unavailability · silent validation rejection · resource
+exhaustion or capacity cliff · ML model or routing failure · data-model or identity defect · filter
+or threshold misconfiguration · unscoped bulk database operation · cascading failure or retry
+amplification · observability defect · analytics correctness defect · dedup failure · authorization
+defect · client or product defect · capacity near-miss
+
+Observability defects are expanded in the next section, and the invariants at the end of this file
+name the class each one points to.
 
 A calibration note: symptoms that originate from a monitor's output or from a prevailing worry are
 the ones that most often turn out not to be real, while content a human actually observed usually is.

From 141b35091ff58bc77a06bfab53f2c53549b853d3 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 14:33:01 -0400
Subject: [PATCH 06/10] newtab/hnt-backend-investigation: prompt for Sentry MCP
 setup when absent

---
 .../skills/hnt-backend-investigation/SKILL.md |  9 +++++++
 .../references/data-sources.md                | 24 +++++++++++++++----
 2 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
index ff9ea09..3f7ebe7 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -64,6 +64,9 @@ the single number or shape you need, and ask them to read it back.
 
 **Then keep investigating the other hypotheses immediately. Do not wait.**
 
+One standing exception to raising a source only when it looks promising: missing Sentry tools, which
+you flag on sight in step 1.
+
 Restate it exactly once more — whichever comes first: they confirm, the unblocked lines run out, or
 you are three probes into a line you had already judged weaker than the blocked one. That is the
 second and last ask. If there is still no response, finish steps 7 and 8 with `Status: blocked`,
@@ -121,6 +124,12 @@ worse than the alert says. When the report is a Sentry alert, resolve it to a co
 volume, and decompose the issue by error message before trusting its title or its trend — the
 alert-quality checks are in `references/failure-modes.md`.
 
+**If the `mcp__sentry__` tools are not available, say so before going any further.** Sentry is the
+primary evidence plane for anything error-shaped in this stack, and steps 1, 2 and 6 all read from
+it, so its absence is not a detail to work around quietly. Raise the setup errand on sight rather
+than weighing it against your other lines; `references/data-sources.md` carries the command. Then
+carry on with the planes you do have.
+
 **If you cannot reproduce it, that is a result, not a blocker.** Record the exact attempt — surface,
 locale, time, request, what you saw instead — then switch the question to why the reporter saw it
 and you do not: a stratum you did not hit, a window that has closed, a cache, a client version, or a
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
index e4da4a9..3f452b8 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
@@ -66,10 +66,23 @@ item you trace rather than assuming it.
 
 ## Sentry
 
-Reachable through the Sentry MCP server if one is connected (its tools are prefixed `mcp__sentry__`).
-Prefer the event-search tool over issue-search when you need a volume breakdown by error message —
-issue search alone will not decompose an umbrella fingerprint. If no Sentry tools are present, treat
-it as a blocked source.
+Reachable through the Sentry MCP server (its tools are prefixed `mcp__sentry__`). Prefer the
+event-search tool over issue-search when you need a volume breakdown by error message — issue search
+alone will not decompose an umbrella fingerprint.
+
+**If no `mcp__sentry__` tools are available, the server is not set up in this session, and you should
+say so immediately.** Do not weigh this one against your other lines first; most symptoms here are
+error-shaped, and steps 1, 2 and 6 all read from Sentry, so an investigation without it is working
+half-blind. Raise it as a `DEV:` task with the command:
+
+```
+! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp
+```
+
+followed by `/mcp` to authenticate in the browser. Servers load when a session starts, so the tools
+may not appear until Claude Code is restarted. Keep going meanwhile: service logs and the BigQuery log
+sink cover part of the same ground, and the developer can read event counts off the Sentry web UI for
+you in the interim.
 
 The HNT services are in the **`mozilla`** organisation, prefixed `hnt-`; issue short-ids look like
 `HNT-CRAWL-9`.
@@ -254,7 +267,8 @@ Each is a short errand: give the developer the command, not a description of the
 | BigQuery permission denied on a dataset | Confirm which billing project to use, or request read access to the dataset |
 | `gcloud` lacks access to a Merino project | Request viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer |
 | Zyte key missing from the environment | Export it in the shell they launched from and restart the session, or run the one extraction you need and paste back the JSON — the JSON, not the key |
-| Sentry MCP unavailable or scoped too narrowly | Authenticate the Sentry MCP server for the `mozilla` org, or paste the issue's event counts broken down by error message |
+| No `mcp__sentry__` tools at all | Run `! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart |
+| Sentry connected but unauthenticated or scoped too narrowly | Run `/mcp` and authenticate for the `mozilla` org, or read back the issue's event counts broken down by error message |
 | Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time |
 | The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for |
 

From 7318b1a67800f17db68c392e7bb48b95bcbff166 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 14:34:26 -0400
Subject: [PATCH 07/10] newtab/hnt-backend-investigation: lead gcloud access
 with install and auth

---
 .../references/data-sources.md                     | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
index 3f452b8..12e5e2a 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
@@ -134,8 +134,12 @@ partition column and filter on it, aggregate inside the query rather than pullin
 
 ## BigQuery
 
-Confirm the billing project with a `--dry_run` before the first real query; a personal sandbox is
-usually `moz-fx-dev-<ldap>-sandbox`. If you cannot confirm one, raise it as an access task.
+`bq` ships with the Google Cloud SDK and runs on your `gcloud` credentials, so a missing binary or an
+unauthenticated session is the first thing to rule out, ahead of any dataset permission: `gcloud auth
+list` shows whether there is an active account. The same credentials cover the `gcloud logging` and
+GCS reads below. Confirm the billing project with a `--dry_run` before the first real query; a
+personal sandbox is usually `moz-fx-dev-<ldap>-sandbox`. Anything missing there is an access task, and
+authenticating is interactive so it belongs to the developer.
 
 | Table | What it is for |
 |---|---|
@@ -264,8 +268,10 @@ Each is a short errand: give the developer the command, not a description of the
 | No read-only MySQL login path configured | Set one up, or give you the host and read-only user to configure |
 | AWS SSO session expired | Run `! aws --profile <profile> sso login` in-session, so the output lands here |
 | No AWS profile at all | Say which read-only profile they have, or request one for the account you need |
-| BigQuery permission denied on a dataset | Confirm which billing project to use, or request read access to the dataset |
-| `gcloud` lacks access to a Merino project | Request viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer |
+| `gcloud` or `bq` not installed | Install the Google Cloud SDK, which provides both |
+| `gcloud` installed but not authenticated | Run `! gcloud auth login`, and `! gcloud auth application-default login` as well if you need the Python client libraries |
+| No billing project configured | Name one they can bill, usually `moz-fx-dev-<ldap>-sandbox`, or set it with `gcloud config set project <id>` |
+| Permission denied on a dataset or a Merino project | Request read access, or viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer |
 | Zyte key missing from the environment | Export it in the shell they launched from and restart the session, or run the one extraction you need and paste back the JSON — the JSON, not the key |
 | No `mcp__sentry__` tools at all | Run `! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart |
 | Sentry connected but unauthenticated or scoped too narrowly | Run `/mcp` and authenticate for the `mozilla` org, or read back the issue's event counts broken down by error message |

From 0ca6cf5d4aef433392b61100975bc4c5ecf7c489 Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 14:45:22 -0400
Subject: [PATCH 08/10] newtab/hnt-backend-investigation: make access requests
 concrete

---
 .../references/data-sources.md                | 20 +++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
index 12e5e2a..c28afe2 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
@@ -230,8 +230,9 @@ SSO sessions expire and the login is interactive, so it has to be the developer:
 ## Zyte
 
 Two separate APIs, both **metered — every call costs money**. Save all responses; never bulk-crawl to
-satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`, and reference it as
-`$ZYTE_API_KEY` in anything you save, so no key is written into a query file or the transcript.
+satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Keys are created at
+https://app.zyte.com/o/612928/zyte-api/api-access. Reference the key as `$ZYTE_API_KEY` in anything
+you save, so its value never lands in a query file or the transcript.
 
 **Extraction API** (`ZYTE_API_KEY`) reproduces what the crawler saw for a URL. Request `article` for
 a single page or `articleList` for an index page — not both — and put `extractFrom:
@@ -245,9 +246,8 @@ unapproved domains into the corpus.
 
 **Stats API** (`ZYTE_SECRET_KEY`, `https://zyte-api-stats.zyte.com/api/stats`) is the vendor's own
 view of your traffic: per-domain response-code distribution over time. `organization_id` is
-**required** and auth is HTTP basic with the secret key as the username and an empty password, so a
-bare GET on the endpoint fails validation — take the org id from the production caller
-(`scripts/fetch/zyte_stats.py` in `content-ml-services`) rather than guessing. Only `groupby_time`
+**required** and is `612928`; auth is HTTP basic with the secret key as the username and an empty
+password, so a bare GET on the endpoint fails validation. Only `groupby_time`
 and `groupby_domain` group; `response_codes` is a *filter*, not a grouping, and
 `include_domain_health=true` is rejected without `groupby_domain=true`. This is the right source for
 "did this domain start failing, and when" — your own logs will not show it if the pipeline discards
@@ -264,15 +264,15 @@ Each is a short errand: give the developer the command, not a description of the
 
 | Blocked source | Ask them to |
 |---|---|
-| Corpus MySQL hangs rather than erroring | Connect to VPN. If it still hangs, ask whether the read-only login path is still valid |
-| No read-only MySQL login path configured | Set one up, or give you the host and read-only user to configure |
-| AWS SSO session expired | Run `! aws --profile <profile> sso login` in-session, so the output lands here |
-| No AWS profile at all | Say which read-only profile they have, or request one for the account you need |
+| Corpus MySQL hangs rather than erroring | Connect to Mozilla VPN, then say so; if it still hangs the login path itself is stale |
+| No read-only MySQL login path configured | Create one: `mysql_config_editor set --login-path=prod-curated-corpus-api-readonly --host=<host> --user=<user> --password`, and tell you the path name |
+| AWS SSO session expired | Run `! aws --profile <profile> sso login`, so the output lands here |
+| No AWS profile at all | Run `! aws configure sso` for a read-only role, or name a profile already in their `~/.aws/config` |
 | `gcloud` or `bq` not installed | Install the Google Cloud SDK, which provides both |
 | `gcloud` installed but not authenticated | Run `! gcloud auth login`, and `! gcloud auth application-default login` as well if you need the Python client libraries |
 | No billing project configured | Name one they can bill, usually `moz-fx-dev-<ldap>-sandbox`, or set it with `gcloud config set project <id>` |
 | Permission denied on a dataset or a Merino project | Request read access, or viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer |
-| Zyte key missing from the environment | Export it in the shell they launched from and restart the session, or run the one extraction you need and paste back the JSON — the JSON, not the key |
+| Zyte key missing from the environment | Create one at https://app.zyte.com/o/612928/zyte-api/api-access, `export ZYTE_API_KEY=<key>` in the shell they launch from, and restart the session. Or have them run the single extraction and paste back the JSON, not the key |
 | No `mcp__sentry__` tools at all | Run `! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart |
 | Sentry connected but unauthenticated or scoped too narrowly | Run `/mcp` and authenticate for the `mozilla` org, or read back the issue's event counts broken down by error message |
 | Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time |

From f2e1adc19d46a79c2bf9d5da112849f96a4a637f Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 14:54:37 -0400
Subject: [PATCH 09/10] newtab/hnt-backend-investigation: ask for and remember
 the investigations root

---
 .../skills/hnt-backend-investigation/SKILL.md | 22 ++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
index 3f7ebe7..623506c 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -79,9 +79,25 @@ that was never made.
 
 ## Workspace
 
-One directory holds the whole investigation: `~/investigation/<YYYY>-<mon><DD>-<slug>`, e.g.
-`~/investigation/2026-jul29-empty-de-sections`. Reuse a directory only when its FINDINGS.md is about
-the same symptom. Say which directory you chose; do not ask.
+Each investigation gets its own subdirectory inside one root directory that the developer keeps their
+investigations in. Do not assume where that root is. Establish it in this order:
+
+1. **Look in user memory.** `~/.claude/CLAUDE.md` is where the root is recorded, because it is the
+   only memory scope that applies across every repository you might be launched from. If it names a
+   root, use it and do not ask.
+2. **Otherwise ask, once, before your first write.** Where the developer keeps investigations cannot
+   be derived, and guessing puts files somewhere they did not choose. Ask it as a single question at
+   the start, and run the step-1 probes while you wait rather than idling.
+3. **Then record it.** Append one line to `~/.claude/CLAUDE.md`, creating that file if it does not
+   exist: `Investigations live in <path> (one subdirectory per investigation).` Tell the developer
+   you saved it, so they know they will not be asked again. Auto memory is the wrong home for this:
+   it is keyed to the repository you were launched from, and investigations get started from
+   whichever service repo is to hand.
+
+Then create this investigation's subdirectory inside that root. If the root already holds
+investigations, follow the naming they use; otherwise `<mon><DD>-<slug>`, e.g.
+`jul29-empty-de-sections`. Say which directory you created. Reuse an existing one only when its
+FINDINGS.md is about the same symptom.
 
 - `FINDINGS.md` — one living document. Create it from the step-7 skeleton **before your first
   probe**; steps 2, 4 and 6 write into it as they go, so step 7 is a final pass over a document that

From 346a75af3493867a0744302bf56a52ba13d5240e Mon Sep 17 00:00:00 2001
From: Matt Miermans <m.miermans@gmail.com>
Date: Thu, 30 Jul 2026 14:59:10 -0400
Subject: [PATCH 10/10] newtab/hnt-backend-investigation: add Slack MCP and
 channel posting rule

---
 plugins/newtab/README.md                      |  1 +
 .../skills/hnt-backend-investigation/SKILL.md | 22 +++++++++++++++----
 .../references/data-sources.md                | 22 ++++++++++++++++---
 3 files changed, 38 insertions(+), 7 deletions(-)

diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md
index 9674925..5feec90 100644
--- a/plugins/newtab/README.md
+++ b/plugins/newtab/README.md
@@ -24,5 +24,6 @@ Use this skill when a Sentry alert fires, an editor reports something broken, or
 - a MySQL client, plus Mozilla VPN
 - `uv` or `python3`
 - the Sentry MCP server
+- the Slack MCP server, for `#hnt-dev-be-alerts`
 
 Anything unreachable is raised as a task and reported under "Could not measure".
diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
index 623506c..ebd5ed0 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -54,8 +54,9 @@ pick the sensible option, say which one you picked, and keep going. Ask a blocki
 when proceeding on any assumption would waste the whole investigation.
 
 When a source you cannot reach looks promising, raise it **once** as a pending task carrying the
-whole errand in its own text, prefixed `DEV:` so it reads as theirs rather than yours — e.g.
-`DEV: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at all`.
+whole errand in its own text, prefixed `User action:` so it reads as theirs rather than yours — e.g.
+`User action: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at
+all`.
 Give the exact command, not a description of the problem; for an interactive login use the `!`
 in-session form so the output lands here, e.g.
 `! aws --profile <profile-you-found-in-~/.aws/config> sso login`. If the answer lives in a dashboard
@@ -156,7 +157,8 @@ A report relayed from an editor is ambiguous between what editors see in curatio
 what clients see on the surface. Probe both in parallel and state which population you confirmed.
 
 The moment you confirm client-visible impact that is still happening, state it in one line — what,
-how big, since when — and keep investigating.
+how big, since when — and keep investigating. If that deserves a heads-up in `#hnt-dev-be-alerts`,
+follow the posting rule at the end of step 8.
 
 ## Step 2 — establish what "normal" is
 
@@ -172,7 +174,8 @@ your own inference. Build that prior yourself before asking for it — most of i
   model or config artifacts.
 - **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of
   week. Do not ask; compute it.
-- **Whether it has happened before** — search Sentry for the same signature, and the service repos'
+- **Whether it has happened before** — search Sentry for the same signature, recent alerts in
+  `#hnt-dev-be-alerts`, and the service repos'
   GitHub issues.
 
 What is left is genuinely in the developer's head: anything in flight that a repo or a dashboard
@@ -319,3 +322,14 @@ Any telemetry plane you could not reach, and the access that would unblock it.
 - **Leave the follow-up as a task**, owned by whoever will act: confirm the fix shipped and the
   metric actually recovered. State that you have done so rather than asking whether to.
 - Hand back a two-line verdict plus the FINDINGS.md path. Do not paste the document into chat.
+
+### Posting to `#hnt-dev-be-alerts`
+
+Updates worth sharing go to `#hnt-dev-be-alerts`. Search the channel first: if the investigation
+started from an alert or message posted there in the last day or two, reply **in that thread**, so the
+diagnosis stays attached to the alert people already saw. Only start a new thread when nothing recent
+relates.
+
+**Ask before every single message, showing the exact text and where it will go.** Approval for one
+post is not approval for the next, and this is a shared channel colleagues act on. If approval does
+not come, keep the draft in FINDINGS.md and carry on; an unposted update is not a reason to stop.
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
index c28afe2..1ced8f1 100644
--- a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
@@ -7,6 +7,7 @@ Reference for the `hnt-backend-investigation` skill.
 - The pipeline, and which repo owns each stage
 - Vocabulary: surface, locale, section, and the id to trace items by
 - Sentry — projects and the traps in reading them
+- Slack — `#hnt-dev-be-alerts`, as alert history and as where updates go
 - Merino — live requests and GCP projects
 - BigQuery — the tables worth knowing and the traps in reading them
 - Curated corpus MySQL — access and schema shape
@@ -73,10 +74,10 @@ alone will not decompose an umbrella fingerprint.
 **If no `mcp__sentry__` tools are available, the server is not set up in this session, and you should
 say so immediately.** Do not weigh this one against your other lines first; most symptoms here are
 error-shaped, and steps 1, 2 and 6 all read from Sentry, so an investigation without it is working
-half-blind. Raise it as a `DEV:` task with the command:
+half-blind. Raise it as a `User action:` task with the command:
 
 ```
-! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp
+! claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp
 ```
 
 followed by `/mcp` to authenticate in the browser. Servers load when a session starts, so the tools
@@ -106,6 +107,20 @@ Traps:
 - Absence of events is weak evidence: a process that dies at startup, or a job that drops items
   while reporting success, emits nothing.
 
+## Slack
+
+`#hnt-dev-be-alerts` carries the backend alerts for this stack, which makes it two things at once: a
+record of what has already fired, worth searching in step 2 before you conclude something is new, and
+the place investigation updates go. Reachable through the Slack MCP server, whose tools are prefixed
+`mcp__slack__`:
+
+```
+! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp
+```
+
+Reading it is free. Posting needs the developer's approval for **each** message, and prefers a reply
+in the existing alert thread over a new one: the rule is at the end of step 8 in SKILL.md.
+
 ## Merino
 
 Reproducing the client call is often the fastest confirmation of a client-visible symptom:
@@ -273,8 +288,9 @@ Each is a short errand: give the developer the command, not a description of the
 | No billing project configured | Name one they can bill, usually `moz-fx-dev-<ldap>-sandbox`, or set it with `gcloud config set project <id>` |
 | Permission denied on a dataset or a Merino project | Request read access, or viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer |
 | Zyte key missing from the environment | Create one at https://app.zyte.com/o/612928/zyte-api/api-access, `export ZYTE_API_KEY=<key>` in the shell they launch from, and restart the session. Or have them run the single extraction and paste back the JSON, not the key |
-| No `mcp__sentry__` tools at all | Run `! claude mcp add --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart |
+| No `mcp__sentry__` tools at all | Run `! claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart |
 | Sentry connected but unauthenticated or scoped too narrowly | Run `/mcp` and authenticate for the `mozilla` org, or read back the issue's event counts broken down by error message |
+| No `mcp__slack__` tools, and an update needs posting | Run `! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp`, then `/mcp` to authenticate; or post the drafted message to `#hnt-dev-be-alerts` themselves |
 | Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time |
 | The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for |