diff --git a/plugins/newtab/README.md b/plugins/newtab/README.md
index 407457f..5feec90 100644
--- a/plugins/newtab/README.md
+++ b/plugins/newtab/README.md
@@ -9,3 +9,21 @@ Skills and tools for Firefox newtab development.
 Adds `@nova-cleanup` comments to Firefox newtab code for Project Nova parallel implementation tracking.
 
 Use this skill when adding cleanup comments to Nova-related changes. See [`skills/nova-cleanup-comments/SKILL.md`](skills/nova-cleanup-comments/SKILL.md) for details.
+
+### `hnt-backend-investigation`
+
+Diagnostic workflow for Home New Tab **backend** incidents, covering Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Its output is an evidence-backed `FINDINGS.md` carrying a root cause and quantified impact.
+
+Use this skill when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale. See [`skills/hnt-backend-investigation/SKILL.md`](skills/hnt-backend-investigation/SKILL.md) for details.
+
+**Optional tools:**
+
+- `gh`
+- `gcloud` and `bq`
+- the `aws` CLI, with a read-only SSO profile
+- a MySQL client, plus Mozilla VPN
+- `uv` or `python3`
+- the Sentry MCP server
+- the Slack MCP server, for `#hnt-dev-be-alerts`
+
+Anything unreachable is raised as a task and reported under "Could not measure".
diff --git a/plugins/newtab/skills/hnt-backend-investigation/SKILL.md b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
new file mode 100644
index 0000000..ebd5ed0
--- /dev/null
+++ b/plugins/newtab/skills/hnt-backend-investigation/SKILL.md
@@ -0,0 +1,335 @@
+---
+name: hnt-backend-investigation
+description: Investigates Home New Tab backend errors, outages, and data-quality problems across Merino, the curated corpus, admin-api, the article crawler, and the ML section pipeline. Confirms the symptom independently, probes competing hypotheses in parallel, stratifies metrics, tries to break its own conclusion, then writes an evidence-backed FINDINGS doc with a root cause and quantified impact. Use when a Sentry alert fires, an editor reports something broken, or recommendations look empty, wrong, or stale.
+---
+
+# New Tab (HNT) Backend Investigation
+
+Diagnose a problem in the services behind Firefox Home New Tab recommendations — Merino, the
+curated corpus, admin-api, the article crawler, the ML section pipeline, the data pipelines. Not
+for in-tree `browser/extensions/newtab` frontend bugs.
+
+Your product is a **diagnosis**: a root cause, the evidence for it, and the impact quantified. Two
+populations can be affected, and every finding should say which: **Firefox New Tab clients** (everyone who sees recommendations) and **editors/curators**
+(the internal editorial team working through curation-admin-tools and admin-api).
+
+**How to reach each system, and the traps that will silently give you wrong answers** — read before
+your first probe in step 1: [references/data-sources.md](references/data-sources.md)
+
+**How these systems fail, the invariants to check, and falsification moves that work** — read when a
+probe comes back empty, when a story starts looking too clean, or when your hypothesis list has
+narrowed to one: [references/failure-modes.md](references/failure-modes.md)
+
+## How to think about this
+
+Four habits do most of the work. They matter more than any specific query below.
+
+**Confirm the symptom before explaining it.** Independently reproduce or measure the reported
+problem first, whatever its source. An alert can be miscalibrated, mis-scoped, or watching the
+wrong layer, and a description passed through two people can drift. Until you have seen the
+symptom yourself in data you pulled, you do not know what you are diagnosing.
+
+**Form the hypothesis before you run the check.** Write down what you expect to see, and what you
+would expect to see if the hypothesis were false. A query designed to confirm and a query designed
+to discriminate look similar and are not. The characteristic failure in this domain is a correct
+query paired with a wrong inference, stated confidently.
+
+**Work several lines at once.** When more than one explanation is plausible, probe them in parallel
+rather than following the most attractive one to its end. Breadth is cheap in the unmetered planes —
+Sentry, repo reads, CloudWatch alarm history, one live Merino request — and tunnel vision is the
+expensive failure. It is not free in BigQuery or Zyte: dry-run and select narrow columns before you
+fan out. Keep a list of every line you considered and record which you dropped and why.
+
+**Prefer measuring to reasoning.** Reproduce the request. Run the model on real inputs. Count the
+actual rows. Read the code path instead of assuming its behaviour. A plausible mechanism becomes a
+number, and only numbers survive review.
+
+Write like the evidence is asymmetric, because it usually is. Avoid "always", "never", "in every
+case"; say what each count is a count of, and whether it is a floor, a ceiling, or a sample.
+
+## Don't stall on the developer
+
+Every pause costs wall-clock, because the developer may not be watching. Default to proceeding:
+pick the sensible option, say which one you picked, and keep going. Ask a blocking question only
+when proceeding on any assumption would waste the whole investigation.
+
+When a source you cannot reach looks promising, raise it **once** as a pending task carrying the
+whole errand in its own text, prefixed `User action:` so it reads as theirs rather than yours — e.g.
+`User action: connect to VPN so corpus MySQL is reachable — settles whether the DE items exist at
+all`.
+Give the exact command, not a description of the problem; for an interactive login use the `!`
+in-session form so the output lands here, e.g.
+`! aws --profile <profile-you-found-in-~/.aws/config> sso login`. If the answer lives in a dashboard
+or console you have no credential for, do not ask for access at all — name the view, the filter, and
+the single number or shape you need, and ask them to read it back.
+
+**Then keep investigating the other hypotheses immediately. Do not wait.**
+
+One standing exception to raising a source only when it looks promising: missing Sentry tools, which
+you flag on sight in step 1.
+
+Restate it exactly once more — whichever comes first: they confirm, the unblocked lines run out, or
+you are three probes into a line you had already judged weaker than the blocked one. That is the
+second and last ask. If there is still no response, finish steps 7 and 8 with `Status: blocked`,
+name the one unblock under "Could not measure", and leave the access task pending.
+
+Stopping outright is a last resort: only when the blocked source is the only thing that can settle
+the question **and** it cannot be reduced to a question they can answer for you — a dashboard,
+console, or explore is almost never a real stop. A request that quietly disappears is worse than one
+that was never made.
+
+## Workspace
+
+Each investigation gets its own subdirectory inside one root directory that the developer keeps their
+investigations in. Do not assume where that root is. Establish it in this order:
+
+1. **Look in user memory.** `~/.claude/CLAUDE.md` is where the root is recorded, because it is the
+   only memory scope that applies across every repository you might be launched from. If it names a
+   root, use it and do not ask.
+2. **Otherwise ask, once, before your first write.** Where the developer keeps investigations cannot
+   be derived, and guessing puts files somewhere they did not choose. Ask it as a single question at
+   the start, and run the step-1 probes while you wait rather than idling.
+3. **Then record it.** Append one line to `~/.claude/CLAUDE.md`, creating that file if it does not
+   exist: `Investigations live in <path> (one subdirectory per investigation).` Tell the developer
+   you saved it, so they know they will not be asked again. Auto memory is the wrong home for this:
+   it is keyed to the repository you were launched from, and investigations get started from
+   whichever service repo is to hand.
+
+Then create this investigation's subdirectory inside that root. If the root already holds
+investigations, follow the naming they use; otherwise `<mon><DD>-<slug>`, e.g.
+`jul29-empty-de-sections`. Say which directory you created. Reuse an existing one only when its
+FINDINGS.md is about the same symptom.
+
+- `FINDINGS.md` — one living document. Create it from the step-7 skeleton **before your first
+  probe**; steps 2, 4 and 6 write into it as they go, so step 7 is a final pass over a document that
+  already exists. Rewrite it in place, not as `FINDINGS-v2.md`.
+- `queries/` and `results/` — every query as a file, its output alongside under the same basename.
+- `api_responses/` — raw JSON from live calls. Save them even when they look boring; metered APIs
+  cost money to re-hit and the data may be gone tomorrow.
+- Aggregate before saving. Counts, rates, and shapes answer nearly every question these
+  investigations ask; this directory sits outside any repo and gets linked into tickets, so prefer a
+  distribution over a dump of rows carrying editor identities or user data.
+- Rewrite only the FINDINGS.md you wrote and delete only files this session created — anything else
+  in the directory belongs to another investigation.
+- Python work goes in an isolated venv inside this directory: `uv venv` if `uv` is installed,
+  otherwise `python3 -m venv`.
+
+## Step 1 — pin down the report
+
+Establish these five facts from the data. The alert or issue carries the symptom and its volume,
+stratifying gives you the surface and locale, and a live request tells you whether it is still
+happening.
+
+| Fact | Why it matters |
+|---|---|
+| Exact symptom, verbatim | "Empty section" and "wrong items in section" have disjoint causes |
+| Who noticed, and how | Alert / editor report / spotted by hand — sets what evidence exists |
+| First and last seen, **with timezone** | Anchors the timeline; reports usually arrive late |
+| Surface, locale, section, client version | The stratum is very often the diagnosis |
+| Still happening right now? | Live reproduction vs. historical forensics |
+
+**If it is still happening, capture the perishable evidence first** — a live
+`curated-recommendations` request, current logs, current per-section counts — and save the raw
+responses under `api_responses/`. Build the step-2 prior while those probes are in flight; history
+keeps, live signal does not.
+
+**An error-rate signal is not yet a symptom.** Measure the erroring stage's *output* to fix the
+blast radius — items per section and freshest item timestamp per surface for assembly, a live Merino
+request for anything client-facing — so you can say whether Firefox clients are affected at all. An
+error floor that never reaches the output is a not-incident; errors flat with output at zero is
+worse than the alert says. When the report is a Sentry alert, resolve it to a concrete error and a
+volume, and decompose the issue by error message before trusting its title or its trend — the
+alert-quality checks are in `references/failure-modes.md`.
+
+**If the `mcp__sentry__` tools are not available, say so before going any further.** Sentry is the
+primary evidence plane for anything error-shaped in this stack, and steps 1, 2 and 6 all read from
+it, so its absence is not a detail to work around quietly. Raise the setup errand on sight rather
+than weighing it against your other lines; `references/data-sources.md` carries the command. Then
+carry on with the planes you do have.
+
+**If you cannot reproduce it, that is a result, not a blocker.** Record the exact attempt — surface,
+locale, time, request, what you saw instead — then switch the question to why the reporter saw it
+and you do not: a stratum you did not hit, a window that has closed, a cache, a client version, or a
+monitor measuring something other than what it claims. Classify as `not-incident` only after that
+second question has an answer; unconfirmed is not refuted.
+
+A report relayed from an editor is ambiguous between what editors see in curation-admin-tools and
+what clients see on the surface. Probe both in parallel and state which population you confirmed.
+
+The moment you confirm client-visible impact that is still happening, state it in one line — what,
+how big, since when — and keep investigating. If that deserves a heads-up in `#hnt-dev-be-alerts`,
+follow the posting rule at the end of step 8.
+
+## Step 2 — establish what "normal" is
+
+You do not know what is currently normal in this system, so you have no basis on which to reject
+your own inference. Build that prior yourself before asking for it — most of it is in reach:
+
+- **Which surfaces are serving** — surfaces present and not disabled in the section data, or a live
+  Merino request for the locale in question.
+- **Whether the crawl is feeding them** — per-surface, per-source volume over the last day from the
+  crawl data. These two sets differ, and a crawled surface with no sections is not automatically a
+  bug.
+- **What shipped recently** — recent commits in the service repos, releases in Sentry, timestamps on
+  model or config artifacts.
+- **Whether this is normally noisy or seasonal** — a trailing profile of the same metric, by day of
+  week. Do not ask; compute it.
+- **Whether it has happened before** — search Sentry for the same signature, recent alerts in
+  `#hnt-dev-be-alerts`, and the service repos'
+  GitHub issues.
+
+What is left is genuinely in the developer's head: anything in flight that a repo or a dashboard
+would not show, and whether a postmortem or incident register already covers this area. Raise that
+as one short note and carry on without waiting; an existing write-up can answer in a paragraph what
+costs hours to reconstruct.
+
+Record what you derive and what they tell you in FINDINGS.md, keeping the two apart. When a later
+query contradicts the prior, treat your own inference as the suspect first, and say so.
+
+## Step 3 — form hypotheses from the data, then probe them in parallel
+
+Hypotheses come from what you have already measured, not from a catalogue. Start where step 1 put
+you: the stage whose output is wrong, the stratum that is affected, the moment it changed. Ask what
+could produce exactly that, follow the data one hop upstream, and let each result generate the next
+question. A hypothesis you cannot tie to something you have observed is a guess competing for the
+same probe budget as one you can.
+
+Work the live ones in parallel rather than serially. Before firing, write beside each hypothesis the
+result that would kill it. Then issue the probes as independent tool calls in a single batch,
+backgrounding anything slow and handing a line that needs several dependent steps to a subagent so
+the batch still returns together. Cap the first wave at four or five cheap, independent probes, and
+hold metered calls and large scans for the second wave. Name each `queries/` and `results/` pair
+after its hypothesis, so a result cannot be attributed to the wrong line.
+
+Then look at what came back and do it again. Each round should either kill a line or sharpen the next
+question; when a round kills everything, the data has moved you upstream rather than left you stuck.
+If you are down to a single surviving hypothesis, that is the moment to check the failure classes and
+silent-failure mechanisms in `references/failure-modes.md` against what you have measured, as a guard
+against having narrowed too early.
+
+Two moves consistently break cases open: **leave the tool you started in** — when a source stops
+yielding, measure the same thing somewhere else, since the failure is often invisible in the plane
+you began with — and **compare against sibling strata**, which localises a fault faster than reading
+code. Both are expanded in `references/failure-modes.md`.
+
+## Step 4 — build the timeline
+
+An explicit `timestamp (UTC) | observation | source` table.
+
+- Normalise every timestamp to UTC and label it. Timezone mismatch is a common source of phantom
+  gaps, and a subtraction error can manufacture an outage that never happened.
+- Extend the window to weeks, not hours — these failures are frequently much older than the report.
+  Widen freely on the cheap planes; on a billed plane, widen only after a narrow window has shown
+  you something.
+- Distinguish *first occurrence* from *first noticed*, and state both.
+- Line the window up against the deploys and config changes from step 2.
+- Treat retention limits as limits: a "first seen" date can be the edge of a retention window rather
+  than onset. See `references/data-sources.md`.
+
+## Step 5 — stratify before concluding anything
+
+An aggregate that looks fine is the normal way these problems hide. Slice every metric by
+**domain, locale, region, surface, section, experiment branch, and client/addon version**, and look
+for a single stratum at zero or down sharply against its own trailing median rather than against
+yesterday. When the report says "some users" with no stratum attached, treat experiment branch,
+region, and rollout state as the first cuts — some surfaces are reachable only through experiment
+enrolment, so an enrolled/unenrolled split is invisible to a locale slice.
+
+Strata are not interchangeable, and the volumes are strongly seasonal; both traps, and the
+per-surface source mix, are detailed in `references/data-sources.md`. Compare each stratum against
+its own history.
+
+**Liveness is not health.** "The job ran successfully" is compatible with total data loss in every
+one of these pipelines. Count what came out, compare it against what went in, and never accept a
+green run as evidence.
+
+If the measurements exonerate the backend, stop at the service boundary. Read far enough to name the
+owning team and the contract that is being broken, then hand it over rather than continuing into a
+system this skill does not cover. A second, unrelated anomaly you trip over on the way is a one-line
+note plus a task, not a second investigation.
+
+## Step 6 — try to break your own conclusion
+
+Before writing anything down as fact, attack it:
+
+- Check the boring explanations first: timezone; a filter in your own query; a `LIMIT` silently
+  truncating; an unrepresentative code path; sampling; a column that does not mean what its name
+  says; a partially-launched feature.
+- Then reach for the falsification moves in `references/failure-modes.md` — the code path, exact
+  enumeration, repetition, the real limit, the config's history — picking the ones that would break
+  your specific mechanism.
+- **Measure the effect, not just the mechanism.** A guard that provably runs is not evidence that
+  the outcome is correct; verify the outcome separately.
+- Cross-check against a second, independent source. One source is a hypothesis.
+- Re-read the step-2 prior. If your conclusion implies that something the prior says is working is
+  broken, re-check your own measurement first — a filter in your query, a stratum mismatch, or a
+  wrong surface identifier is the likelier explanation. If the measurement survives that re-check,
+  the measurement wins: state the contradiction explicitly in FINDINGS.md, note it to the developer
+  in one line, and keep going.
+
+Mark every claim **verified**, **inferred**, or **refuted**. Keep the refuted ones in the document;
+they stop the next person re-running them.
+
+## Step 7 — write FINDINGS.md
+
+```markdown
+# <symptom> — investigation
+
+**Status:** investigating | root cause identified | blocked
+**Class:** outage | degradation | data-quality defect | user-facing bug | near-miss | not-incident
+**Severity:** critical | high | medium | low
+**Affected:** Firefox clients | editors | both — with a magnitude, not an adjective
+
+## Summary
+Three sentences: what broke, since when, what the user-visible effect is.
+
+## Timeline (UTC)
+| Time | Observation | Source |
+
+## Prior
+**Derived** — per-surface volumes, recent deploys and model artifacts, trailing profile, prior
+occurrences. Each with the query or link.
+**Reported** — verbatim answers from the developer, or `asked <time UTC>, no answer received`.
+
+## Evidence
+Numbered findings. Each: the claim, the query or command, the result, and
+verified/inferred/refuted. Reference saved files by path — every number reproducible from this
+directory.
+
+## Root cause
+The mechanism, with the code path or config that produces it. "Not established" when it is not.
+
+## Hypotheses considered and dropped
+Every line you enumerated, and what ruled each one out — or why it was not pursued.
+
+## Impact quantified
+Rows, requests, users, hours, locales. A number, or an explicit "unquantified because X".
+
+## Proposed fix
+The change as a diff in this document, at file-and-line where you can get there, produced by
+reading the clones.
+
+## Could not measure
+Any telemetry plane you could not reach, and the access that would unblock it.
+```
+
+## Step 8 — close the loop
+
+- **Report what you could not reach.** Missing dashboard, IAM, token, or VPN access is a finding,
+  not an inconvenience to route around. Leave any unresolved access task on the list rather than
+  clearing it.
+- **Link prior art** — older investigation, ticket, postmortem, or a related error already known.
+- **Leave the follow-up as a task**, owned by whoever will act: confirm the fix shipped and the
+  metric actually recovered. State that you have done so rather than asking whether to.
+- Hand back a two-line verdict plus the FINDINGS.md path. Do not paste the document into chat.
+
+### Posting to `#hnt-dev-be-alerts`
+
+Updates worth sharing go to `#hnt-dev-be-alerts`. Search the channel first: if the investigation
+started from an alert or message posted there in the last day or two, reply **in that thread**, so the
+diagnosis stays attached to the alert people already saw. Only start a new thread when nothing recent
+relates.
+
+**Ask before every single message, showing the exact text and where it will go.** Approval for one
+post is not approval for the next, and this is a shared channel colleagues act on. If approval does
+not come, keep the draft in FINDINGS.md and carry on; an unposted update is not a reason to stop.
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
new file mode 100644
index 0000000..1ced8f1
--- /dev/null
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/data-sources.md
@@ -0,0 +1,298 @@
+# New Tab (HNT) backend — where to look
+
+Reference for the `hnt-backend-investigation` skill.
+
+## Contents
+
+- The pipeline, and which repo owns each stage
+- Vocabulary: surface, locale, section, and the id to trace items by
+- Sentry — projects and the traps in reading them
+- Slack — `#hnt-dev-be-alerts`, as alert history and as where updates go
+- Merino — live requests and GCP projects
+- BigQuery — the tables worth knowing and the traps in reading them
+- Curated corpus MySQL — access and schema shape
+- The editor plane — admin-api and curated-corpus-api
+- AWS — profiles, and what logs give you that Sentry cannot
+- Zyte — extraction API and the vendor's own stats
+- **Access requests** — the one-line ask for each gated source
+
+Write your own queries: this file gives you the shape of the data and the traps, not canned SQL.
+Access varies by developer. Verify a source is reachable before building a plan around it, and if it
+is not, follow "Don't stall on the developer" in SKILL.md — raise it once as a task and keep
+investigating, rather than stopping or silently substituting a weaker source.
+
+## The pipeline
+
+**crawl / discovery → hydration (Zyte) → curated corpus → ML section assembly → curated-corpus-api →
+client-api → Merino → Firefox New Tab.** Editors act on the corpus through curation-admin-tools →
+admin-api → curated-corpus-api. Telemetry lands in BigQuery.
+
+`client-api` is the Apollo federated router (`Pocket/pocket-monorepo`, `servers/client-api`) and it
+is easy to miss: it has **no Sentry project of its own in either organisation**, so a failure there
+surfaces as a Merino symptom. Merino also reaches it at the **prod** endpoint unconditionally —
+`CorpusApiGraphConfig.endpoint` returns `CORPUS_API_PROD_ENDPOINT` regardless of Merino's own
+environment, and the dev constant beside it is unused — so stage Merino reads prod corpus data.
+
+| Repo | GitHub | Role |
+|---|---|---|
+| `merino-py` | mozilla-services/merino-py | Serves New Tab recommendations and Firefox Suggest |
+| `content-monorepo` | Pocket/content-monorepo | Curated corpus, recommendations, section manager |
+| `content-ml-services` | mozilla/content-ml-services | Crawl, classification, section assembly (Metaflow, Cloud Functions) |
+| `pocket-monorepo` | Pocket/pocket-monorepo | client-api federated router, shared infrastructure |
+| `curation-admin-tools` | Pocket/curation-admin-tools | Editor-facing web app |
+| `admin-api` | Pocket/admin-api | Federated GraphQL gateway for the admin tools |
+| `bigquery-etl` | mozilla/bigquery-etl | New Tab engagement and Merino recommendation ETL |
+| `serverless-image-cache` | Pocket/serverless-image-cache | Thumbor image resize and cache |
+| `firefox` | mozilla-firefox/firefox | Client side of the contract (`browser/extensions/newtab`) |
+
+Locate clones rather than assuming paths:
+`find ~ -maxdepth 3 -type d -name .git -print0 2>/dev/null | xargs -0 -n1 dirname`. For a repo that
+is not cloned, read it through `gh api` or `gh search code`.
+
+## Vocabulary: surface, locale, section
+
+A **surface** is one locale/market feed of the corpus, written `NEW_TAB_EN_US`. A **section** is a
+topic row inside a surface. Merino's request `locale` is hyphenated (`en-US`) and the surface is
+**derived** from language plus region — the mapping table is `SurfaceId` in
+`merino/curated_recommendations/localization.py`, not a reformatting of the locale string. Some
+surfaces are reachable only through experiment enrolment. Stratifying by locale and stratifying by
+surface are therefore not the same slice.
+
+Merino appends `utm_source=firefox-newtab-<surface>` to item URLs (`get_utm_source` and
+`update_url_utm_source` in `curated_recommendations/corpus_backends/utils.py`), so a URL taken from a
+response will never match a stored `url`. Trace items by the stable id instead: the response's
+`corpusItemId` is the corpus GraphQL item `id`, which lines up with `ApprovedItem.externalId` in
+corpus MySQL and `approved_corpus_item_external_id` in BigQuery. Confirm that last hop on the first
+item you trace rather than assuming it.
+
+## Sentry
+
+Reachable through the Sentry MCP server (its tools are prefixed `mcp__sentry__`). Prefer the
+event-search tool over issue-search when you need a volume breakdown by error message — issue search
+alone will not decompose an umbrella fingerprint.
+
+**If no `mcp__sentry__` tools are available, the server is not set up in this session, and you should
+say so immediately.** Do not weigh this one against your other lines first; most symptoms here are
+error-shaped, and steps 1, 2 and 6 all read from Sentry, so an investigation without it is working
+half-blind. Raise it as a `User action:` task with the command:
+
+```
+! claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp
+```
+
+followed by `/mcp` to authenticate in the browser. Servers load when a session starts, so the tools
+may not appear until Claude Code is restarted. Keep going meanwhile: service logs and the BigQuery log
+sink cover part of the same ground, and the developer can read event counts off the Sentry web UI for
+you in the interim.
+
+The HNT services are in the **`mozilla`** organisation, prefixed `hnt-`; issue short-ids look like
+`HNT-CRAWL-9`.
+
+`hnt-crawl` · `hnt-metaflow` · `hnt-curated-corpus-api` · `hnt-admin-api` ·
+`hnt-curation-admin-tools` · `hnt-section-manager-lambda` · `hnt-corpus-scheduler-lambda` ·
+`hnt-prospect-api` · `hnt-prospect-translation-lambda` · `hnt-serverless-image-cache` ·
+`hnt-braze-content-proxy` · `hnt-feature-flags`
+
+Merino is separate: project `merino-py`, also in `mozilla`. Some older projects exist in the `pocket`
+org; prefer the `hnt-` ones for anything current. Enumerate the current projects before relying on
+this list — a project that does not resolve is a naming change, not evidence of zero errors.
+
+Traps:
+
+- **`first seen` is bounded by retention**, so it is not onset. Nearly every long-running issue
+  reports a first-seen date at the edge of the retention window.
+- Issue volume is dominated by long-standing error floors. Establish what was already there before
+  attributing anything to today, and check whether a floor *changed* rather than whether it exists.
+- Decompose an issue by error message before trusting its title or trending it.
+- Absence of events is weak evidence: a process that dies at startup, or a job that drops items
+  while reporting success, emits nothing.
+
+## Slack
+
+`#hnt-dev-be-alerts` carries the backend alerts for this stack, which makes it two things at once: a
+record of what has already fired, worth searching in step 2 before you conclude something is new, and
+the place investigation updates go. Reachable through the Slack MCP server, whose tools are prefixed
+`mcp__slack__`:
+
+```
+! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp
+```
+
+Reading it is free. Posting needs the developer's approval for **each** message, and prefers a reply
+in the existing alert thread over a new one: the rule is at the end of step 8 in SKILL.md.
+
+## Merino
+
+Reproducing the client call is often the fastest confirmation of a client-visible symptom:
+`POST https://merino.services.mozilla.com/api/v1/curated-recommendations` with a JSON body carrying
+`locale`, `region`, `topics`, `sections`, `feeds` (e.g. `["sections"]`), `enableInterestPicker`, and
+optionally `experimentName` / `experimentBranch` to land on an experiment branch. Send a realistic
+Firefox `User-Agent`. Save every response. First things to check on the payload: section count,
+items per section, presence of `followable` / `allowAds`, and the age of the newest item.
+
+Keep it to the requests you need. If you find yourself writing a loop over locales or branches
+against prod, query the telemetry instead. Stage answers payload-shape questions — but remember it
+reads prod corpus data, so it cannot answer questions about prod corpus content.
+
+GCP projects, for log reads, metrics, and GCS listings — confirm against
+`merino/configs/production.toml` **and `merino/configs/stage.toml`** rather than trusting this table:
+
+| Environment | Project |
+|---|---|
+| Merino prod | `moz-fx-merino-prod-5de4`; some buckets in `moz-fx-merino-prod-1c2f` |
+| Merino stage | `moz-fx-merino-nonprod-ee93`; also `moz-fx-merino-nonprod-db57` |
+
+When the logging API returns nothing useful, the BigQuery log sink for the same project usually does.
+That sink table is far larger than anything in the BigQuery list below — check whether it has a
+partition column and filter on it, aggregate inside the query rather than pulling rows, and
+`--dry_run` first.
+
+## BigQuery
+
+`bq` ships with the Google Cloud SDK and runs on your `gcloud` credentials, so a missing binary or an
+unauthenticated session is the first thing to rule out, ahead of any dataset permission: `gcloud auth
+list` shows whether there is an active account. The same credentials cover the `gcloud logging` and
+GCS reads below. Confirm the billing project with a `--dry_run` before the first real query; a
+personal sandbox is usually `moz-fx-dev-<ldap>-sandbox`. Anything missing there is an access task, and
+authenticating is interactive so it belongs to the developer.
+
+| Table | What it is for |
+|---|---|
+| `moz-fx-mozsoc-ml-prod.prod_rss_news.rss_feed_items` | Articles discovered by crawl — per-surface, per-source (`PAGE` vs `RSS`) volume over time |
+| `moz-fx-mozsoc-ml-prod.prod_articles.zyte_cache` | Hydrated article metadata keyed on `canonical_url` — what the pipeline believes an article says |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.sections_v1` | Section existence, enable/disable state, and surface, as an event stream |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.section_items_v1` | Which items sit in which section, and when each was last touched — the freshness check for a stalled section |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.corpus_items_current_v1` | Current-state view of corpus items — prefer this to reducing the event table yourself |
+| `moz-fx-data-shared-prod.snowflake_migration_derived.approved_corpus_items` / `scheduled_corpus_items` | Maintained current-state views for approved and scheduled items |
+
+Confirm each table still exists before drawing a conclusion from an empty result.
+
+Traps that will silently give you a wrong answer:
+
+- **There is no domain column on `rss_feed_items`.** Derive the domain from `canonical_url` (or
+  `origin_url` for the page it was found on). A per-domain funnel is a computed grouping, not a
+  lookup.
+- **`source` is null for everything crawled before 2025-08-11** — 6.8M of 22.6M rows. A long
+  per-source series will show `PAGE` and `RSS` springing into existence on that date; that is the
+  column being introduced, not the crawl changing.
+- **`crawled_date` and `published_date` are STRING; `crawled_at`, `published_at` and `loaded_at` are
+  TIMESTAMP.** Use the timestamps for any time arithmetic.
+- **Cost does not work the way you expect.** `rss_feed_items` (~23.6 GB) and `zyte_cache` (~128 GB)
+  are **unpartitioned and unclustered**, so a date predicate reduces nothing — only column selection
+  does. Never `SELECT *` on them and always `--dry_run` first. The `snowflake_migration_derived`
+  tables are day-partitioned on `happened_at` but do not require a partition filter, so supply one
+  yourself.
+- **The `*_v1` event tables are event logs, not current state.** Rows accumulate per change, so a
+  plain `COUNT(*)` over-counts. Prefer the current-state views above; where you must reduce the log
+  yourself, take the latest row per id and check the inflation ratio rather than assuming it.
+- **`sections_v1.source` is unusable for current sections** — null on 81% of all sections and on
+  **every** currently-active one. Section ownership (`MANUAL` vs `ML`) comes from
+  `Section.createSource` / `updateSource` / `deactivateSource` in corpus MySQL. A null `source` means
+  unknown, never `MANUAL`.
+- **Surface identifiers differ by table.** Crawl data uses `en_US`; section data uses
+  `NEW_TAB_EN_US`. Joining or comparing them naively produces empty results that look like an outage.
+- **The crawl surface set and the serving surface set are not identical**, so a crawled surface with
+  no sections is not automatically a bug.
+- **Source mix varies by surface** — some have no RSS-sourced content, some no page-crawled content.
+  Judge each surface against its own history.
+- **Volume is strongly day-of-week seasonal.** Compare against a trailing median, never against
+  yesterday.
+- A zero in the crawl table is ambiguous between "found nothing" and "every request failed"; there is
+  no attempt/error record to disambiguate it.
+
+## Curated corpus MySQL
+
+Production database behind curated-corpus-api, reached through a preconfigured read-only login path.
+Check what exists with `mysql_config_editor print --all`, and expect to need VPN — a hang rather than
+an auth error is the usual symptom of being off it.
+
+Useful invocation guards: `--safe-updates` (caps returned rows at 1000 and aborts queries estimated
+to examine over a million) and `SET SESSION max_execution_time=10000` (10s server-side cap). The
+1000-row cap **truncates silently**, so add an explicit `LIMIT` or raise the cap when you need a
+complete set.
+
+Schema `curation_corpus`. The tables that matter: `ApprovedItem` (the corpus itself, keyed by
+`externalId` and `url`), `SectionItem` and `Section` (placement and section config — `Section` also
+carries `createSource` / `updateSource` / `deactivateSource`, the authoritative ML-vs-manual
+ownership), `ScheduledItem` (surface scheduling), `RejectedCuratedCorpusItem`, and the
+`PublisherDomain` / `TrustedDomain` / `ExcludedDomain` domain lists. `SectionItem` runs into the
+millions of rows and `ApprovedItem` into the hundreds of thousands — check indexes with `SHOW INDEX`
+before filtering, since several obvious filter columns are unindexed.
+
+## The editor plane
+
+When the symptom is editor-facing, reproducing it means an authenticated editor session, which is an
+interactive login and therefore the developer's errand rather than yours. What you can do alone:
+`hnt-admin-api` and `hnt-curated-corpus-api` in Sentry for the failure window, the corpus rows the
+mutation would have touched, and the resolver itself in `Pocket/admin-api` and
+`Pocket/content-monorepo`. That is usually enough to name the mechanism without a session.
+
+## AWS
+
+content-monorepo infrastructure runs in AWS. Discover profiles with
+`grep -E '^\[profile' ~/.aws/config` and confirm one works with `sts get-caller-identity`; match prod
+against dev to the environment you are investigating.
+
+CloudWatch Logs Insights is frequently the source that cracks a case — it will give you total
+operation counts and per-error-type breakdowns that Sentry structurally cannot, because Sentry only
+sees what was raised. It bills per GB scanned per query, so pass the narrowest
+`--start-time`/`--end-time` that could answer the question and `stats`-aggregate rather than dumping
+`fields`. Also useful: alarm history (transition timestamps and state-reason margins), and pulling an
+anomaly band itself as a metric-math series to compare its predicted centre against reality.
+
+SSO sessions expire and the login is interactive, so it has to be the developer: see the table below.
+
+## Zyte
+
+Two separate APIs, both **metered — every call costs money**. Save all responses; never bulk-crawl to
+satisfy curiosity. Check for a key with `[ -n "$ZYTE_API_KEY" ] && echo present`. Keys are created at
+https://app.zyte.com/o/612928/zyte-api/api-access. Reference the key as `$ZYTE_API_KEY` in anything
+you save, so its value never lands in a query file or the transcript.
+
+**Extraction API** (`ZYTE_API_KEY`) reproduces what the crawler saw for a URL. Request `article` for
+a single page or `articleList` for an index page — not both — and put `extractFrom:
+"httpResponseBody"` inside `articleOptions` / `articleListOptions` to skip the browser. Check
+**`statusCode` and the extraction probability before anything else**: a non-2xx, a bot wall, or a
+"JavaScript is disabled" page still returns a populated object that reads like success. Probability
+lives at `article.metadata.probability` for a single page and per item at
+`articleList.articles[].metadata.probability` — `articleList.metadata` carries only
+`dateDownloaded`. Compare `canonicalUrl` against the URL you requested; cross-domain canonicals pull
+unapproved domains into the corpus.
+
+**Stats API** (`ZYTE_SECRET_KEY`, `https://zyte-api-stats.zyte.com/api/stats`) is the vendor's own
+view of your traffic: per-domain response-code distribution over time. `organization_id` is
+**required** and is `612928`; auth is HTTP basic with the secret key as the username and an empty
+password, so a bare GET on the endpoint fails validation. Only `groupby_time`
+and `groupby_domain` group; `response_codes` is a *filter*, not a grouping, and
+`include_domain_health=true` is rejected without `groupby_domain=true`. This is the right source for
+"did this domain start failing, and when" — your own logs will not show it if the pipeline discards
+non-allowlisted status codes.
+
+Freshness thresholds are a common thing to want here and a common thing to invent. Derive the
+intended cadence from the assembly schedule in `content-ml-services` and the corpus-scheduler-lambda
+trigger config rather than guessing, and record the interval you used in FINDINGS.md.
+
+## Access requests
+
+Raise one of these only when that source is one of your strongest current lines, then keep working.
+Each is a short errand: give the developer the command, not a description of the problem.
+
+| Blocked source | Ask them to |
+|---|---|
+| Corpus MySQL hangs rather than erroring | Connect to Mozilla VPN, then say so; if it still hangs the login path itself is stale |
+| No read-only MySQL login path configured | Create one: `mysql_config_editor set --login-path=prod-curated-corpus-api-readonly --host=<host> --user=<user> --password`, and tell you the path name |
+| AWS SSO session expired | Run `! aws --profile <profile> sso login`, so the output lands here |
+| No AWS profile at all | Run `! aws configure sso` for a read-only role, or name a profile already in their `~/.aws/config` |
+| `gcloud` or `bq` not installed | Install the Google Cloud SDK, which provides both |
+| `gcloud` installed but not authenticated | Run `! gcloud auth login`, and `! gcloud auth application-default login` as well if you need the Python client libraries |
+| No billing project configured | Name one they can bill, usually `moz-fx-dev-<ldap>-sandbox`, or set it with `gcloud config set project <id>` |
+| Permission denied on a dataset or a Merino project | Request read access, or viewer on the project; say meanwhile whether the question is about payload shape, which stage can answer |
+| Zyte key missing from the environment | Create one at https://app.zyte.com/o/612928/zyte-api/api-access, `export ZYTE_API_KEY=<key>` in the shell they launch from, and restart the session. Or have them run the single extraction and paste back the JSON, not the key |
+| No `mcp__sentry__` tools at all | Run `! claude mcp add --scope user --transport http sentry https://mcp.sentry.dev/mcp`, then `/mcp` to authenticate; the tools appear after a session restart |
+| Sentry connected but unauthenticated or scoped too narrowly | Run `/mcp` and authenticate for the `mozilla` org, or read back the issue's event counts broken down by error message |
+| No `mcp__slack__` tools, and an update needs posting | Run `! claude mcp add --scope user --transport http slack https://mcp.slack.com/mcp`, then `/mcp` to authenticate; or post the drafted message to `#hnt-dev-be-alerts` themselves |
+| Editor-facing symptom needs an authenticated session | Reproduce the click themselves and report the exact error text and time |
+| The answer is in a dashboard you cannot reach | Open it, apply the specific filter you name, and read back the one number or shape you asked for |
+
+Notice the last row: when a dashboard, explore, or console holds the answer, asking for *access* is
+usually the slower path. Asking a precise question about what it shows is faster for both of you.
diff --git a/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
new file mode 100644
index 0000000..d0e8eb3
--- /dev/null
+++ b/plugins/newtab/skills/hnt-backend-investigation/references/failure-modes.md
@@ -0,0 +1,136 @@
+# How New Tab (HNT) backend failures behave
+
+Reference for the `hnt-backend-investigation` skill: the shapes these failures take, the invariants
+to check, and the moves that kill a wrong hypothesis. Read this before you take any hypothesis
+seriously.
+
+## Contents
+
+- Failure classes — the vocabulary, for checking you have not narrowed too early
+- If it came from an alert, audit the alert
+- Why these failures are silent — the mechanisms that turn a failure into a valid-looking result
+- Falsification moves that work — what to reach for when a story feels too clean
+- Structural gaps to report rather than infer past
+- Invariants to check — hard checks, and the class each one points to
+
+## Failure classes
+
+Vocabulary, not a lookup. Do not use this list to guess a cause from a symptom; derive the layer from
+what you have measured. It is here so that when your hypothesis list has narrowed to one or two, you
+can check it against the range of things that actually break in this stack:
+
+vendor extraction-quality defect · upstream unavailability · silent validation rejection · resource
+exhaustion or capacity cliff · ML model or routing failure · data-model or identity defect · filter
+or threshold misconfiguration · unscoped bulk database operation · cascading failure or retry
+amplification · observability defect · analytics correctness defect · dedup failure · authorization
+defect · client or product defect · capacity near-miss
+
+Observability defects are expanded in the next section, and the invariants at the end of this file
+name the class each one points to.
+
+A calibration note: symptoms that originate from a monitor's output or from a prevailing worry are
+the ones that most often turn out not to be real, while content a human actually observed usually is.
+This is a reason to confirm the signal independently in both directions, not a reason to dismiss
+either.
+
+## If it came from an alert, audit the alert
+
+Alert quality is its own failure class, and worth a few minutes before chasing the system. This is an
+addition to confirming the symptom independently, not a substitute for it.
+
+- **Wrong layer** — the alert watches a gateway while the loss happens a hop upstream.
+- **Umbrella fingerprint** — one issue collapses several unrelated failures; decompose by error
+  message before trusting the title or the trend, and anchor on stable issue identifiers rather than
+  numeric ids in the alert body.
+- **Mis-centred or flapping band** — repeated transitions at a fixed offset within the hour, and
+  margins under a percent, mean the band, not the system.
+- **Counting artefact** — a panel that counts per open period rather than per calendar day can be off
+  by an order of magnitude.
+- **Unroutable** — if you cannot tell prod from staging at a glance, fix that first.
+
+## Why these failures are silent
+
+Not under-monitored — structurally unobservable. The recurring mechanism is that an upstream failure
+gets converted into a valid-looking empty or partial result. Expect these:
+
+- A pipeline that keeps only an allowlist of status codes and swallows request errors, so a total
+  block is indistinguishable from "nothing found today".
+- A vendor non-2xx arriving as a partially-filled success object, so a total outage raises nothing.
+- A job reporting invocation-level success while silently dropping every item it received.
+- Input-validation rejections that are logged and never surfaced.
+- An `INNER JOIN` against a frozen upstream catalog shedding rows quietly instead of failing.
+- A process that dies during startup, which emits no error event at all.
+- A bulk write with no row-count guard.
+- An aggregate quality gate passing while one class collapses beneath it.
+- An alarm that treats missing data as "missing" rather than as a breach, sitting against an emitter
+  that publishes nothing at zero — so it goes *quiet* during a total stop.
+
+The consequence for every one of these: **liveness is not health.** Count what came out and compare
+it to what went in.
+
+## Falsification moves that work
+
+Each of these has killed a plausible, confidently-held hypothesis. Reach for them when a story feels
+too clean.
+
+- **Read the code path.** Assumed behaviour of a resolver, filter, or job is the single most common
+  source of a wrong mechanism.
+- **Enumerate the whole population** rather than sampling, when it is free to count — in BigQuery or
+  MySQL, "0 of N" is a different claim from "none in my sample". Against a metered API, sample, say
+  what the sample is, and put any larger enumeration on the task list with its call count.
+- **Repeat the observation N times** before calling anything non-deterministic, then vary one input
+  at a time.
+- **Read the actual limit** — quota, timeout, payload cap, row cap — instead of assuming which one
+  binds. Adjacent limits on the same resource are easy to confuse.
+- **Check the history of the constant or config** you are relying on; a value that used to be true is
+  the classic stale-assumption trap.
+- **Dump the attribute across the entire population** when you suspect truncation or a per-record
+  limit.
+- **Go to the other side's telemetry.** A vendor's or upstream's own status data will show what your
+  logs structurally cannot.
+- **Compare against sibling strata** at the same layer — the healthy peers localise the fault faster
+  than reading code does.
+- **Design a canary that would falsify** the hypothesis rather than a test that would confirm it.
+- **Refute it yourself, in writing.** Before a conclusion leaves FINDINGS.md, draft the strongest
+  case against it under "Hypotheses considered and dropped" and answer that case with a measurement.
+  A second agent handed the evidence and told to break the conclusion is cheaper than waiting for a
+  human reviewer.
+
+## Structural gaps to report rather than infer past
+
+These make certain questions unanswerable from stored data alone. When one blocks you, say so.
+
+- No crawl attempt/success/error record, so a zero row count is ambiguous between "found nothing" and
+  "every request failed".
+- No lifecycle metadata or tombstones on the crawl target list, so an intentional retirement is
+  indistinguishable from an outage.
+- No persisted ML filter funnel, so per-stage drop-off cannot be reconstructed afterwards.
+- No queryable deployment history to line a timeline up against.
+- No status code or extraction-probability stored alongside cached hydration results, so a stored row
+  does not imply a successful extraction.
+
+## Invariants to check
+
+Hard equalities and floors rather than statistical thresholds, so a violation is unambiguous rather
+than a matter of degree. Assert the ones that bear on your hypotheses against the data: a broken one
+localises the fault immediately, and a holding one eliminates a line cheaply.
+
+| Invariant | Class it points to |
+|---|---|
+| Articles/hour > 0 per surface **per source**, judged against that surface's own mix | Upstream unavailability; capacity cliff |
+| Active items per fixed-size section == configured N | Silent validation rejection; model failure |
+| `items_written == items_received` per job invocation | Silent validation rejection |
+| Input-validation reject rate == 0 | Silent validation rejection |
+| Newest active item age per section within that surface's assembly interval | Model or routing failure; capacity cliff |
+| Vendor non-2xx share per domain below a floor | Upstream unavailability; extraction defect |
+| Content-shape ratio per domain within bounds (empty-body %, title length) | Extraction-quality defect |
+| Per-domain funnel positive at every stage: discovered → hydrated → approved | Upstream unavailability; filter misconfiguration |
+| Canonical domain ∈ approved list | Data-model / identity defect |
+| Per-class model recall above a floor before publish | Model or routing failure |
+| Row-count delta on key tables within bounds | Unscoped bulk operation |
+| Every bound parameter in shared SQL supplied by every caller | Analytics correctness |
+| Consumption below documented quota, alerting on the ratio | Capacity near-miss |
+| The alarm itself emits a value at zero | Observability defect |
+
+Stratify anything you assert. A rule that holds globally can be false for one surface, because the
+surfaces do not all draw content the same way.