choiyounggi · choiyounggi · Sep 6, 2026
@@ -104,3 +104,14 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-08-30] revise | frontend/design/design-canvas-workflow — agent-gate Check 3 remediation (PR #164). The gate's fact finding was half right: its "no skill named `design` exists" conclusion is refuted by ground truth (the skill is listed in the authoring session's available-skills roster as `design: Create a design canvas…` and its bundled payload exists on the authoring machine's disk; the reviewer's CI environment lacks the preview, and it conflated the skill with the unrelated `/design-sync` component-upload tool) — but the verifiability objection stands: the skill is an early research preview absent from public docs, so a single un-fetchable in-session source cannot carry `confidence: verified`. Fixed per AGENTS.md: confidence downgraded to field-tested with the experience context stated in the body; two live-fetched public sources added (anthropic.com/news/claude-design-anthropic-labs — official Claude Design product announcement, research preview, Claude Code handoff; explainx.ai 2026 post — /design ships in Claude Code as a research-preview command producing editable artboards via the Artifacts runtime); the mandatory-routing directive and the frontend/index.md load-when line conditioned on the skill appearing in the session's available-skills list, with the existing no-skill edge case as the explicit fallback; the unverifiable "design-review skill" name-drop in the audit edge case generalized to session-provided audit tooling.
 ## [2026-09-03] ingest | databases/selection — new category: datastore selection by workload (4 pages: choosing-a-datastore-by-workload, relational-jsonb-vs-document-store, vector-search-engine-selection, graph-workloads-relational-vs-graph-db); brave-search research, sources cited per page
 ## [2026-09-04] ingest | infrastructure/agent-orchestration/code-graph-as-orientation-layer (field-tested — 1 new page). A locally built code knowledge graph (graphify) is a freshness-gated, lead-not-evidence orientation layer for planning and parallel task decomposition: gate on graph mtime vs git log, symbol-anchored explain/path only, CLI delivery with bounded output, graph-derived assumptions named in reports. Sources: Developers Digest, AQ Score, Autonoma, Tiare Balbi, graphify README + measurements on graphifyy 0.4.23.
+## [2026-09-06] ingest | infrastructure-ci-cd-unparseable-workflow-file — a column-0 line inside `run: |` breaks the YAML; GitHub lists the file by path, registers no triggers, and reports nothing (yaml spec + gh manual + groundwork #16 + local Psych repro)
+## [2026-09-06] ingest | infrastructure-ci-cd-workflow-authored-pull-requests — the four settings behind a workflow that opens and merges its own PR: `can_approve_pull_request_reviews`, personal-repo ruleset without Integration bypass, PAT instead of GITHUB_TOKEN so checks run, `--auto` + `delete_branch_on_merge` (GitHub docs + gh manual + cli/cli#9073 + groundwork PRs #18–#27)
+## [2026-09-06] ingest | backend-common-integrations-contact-details-from-scraped-pages — turn `onlyMainContent` off and read `tel:`/messenger hrefs before adding JS rendering (Firecrawl docs + 30-site measurement)
+## [2026-09-06] ingest | testing-data-testcontainers-python-community-namespace — import service containers from `testcontainers.community.<module>` on 4.15.0+; same class and constructor (upstream shim source + commit ab6cca8 + fresh-venv repro)
+## [2026-09-06] ingest | frontend-design-pointer-attracted-particle-fields — a fractional per-frame pull is a contraction; require release + inner-radius clamp and a many-step distance-floor test (Banach + MDN + cover-letter review F1)
+## [2026-09-06] ingest | testing-mocking-producer-wire-format-in-mocks — lift an id's wire shape from the producer's tests, map the opaque token through the binding message (Google ToT + Fowler ContractTest + Pact + linkly-crew #10)
+## [2026-09-06] ingest | platforms-toolchains-compiler-sysroot-on-macos — reconciled contradiction: Homebrew clang's `etc/clang/<triple>.cfg` pins `-isysroot`, so `SDKROOT` is ignored there; `-isysroot` or `--no-default-config` (clang UsersManual + llvm.rb + local repro on clang 22.1.8)
+## [2026-09-06] ingest | backend-common-llm-binding-instructions-for-agents — "ask the user" binds only when the question tool is named and a transcript-reading PreToolUse gate checks it (Claude Code hooks doc + dev-loop #144)
+## [2026-09-06] ingest | debugging-methodology-reproduce-first — reproduce a script bug under the script's own interpreter (aliases/rc files differ), and check an existing green test's assertion boundary against the symptom (bash/zsh manuals + dev-loop #145 + linkly-crew #10)
+## [2026-09-06] ingest | qa-process-completion-claims — "the stage is healthy, its test passes" needs an assertion past the symptom (linkly-crew #10)
+## [2026-09-06] ingest | testing-mocking-what-to-mock — edge row pointing to producer-wire-format-in-mocks
@@ -0,0 +1,66 @@
+---
+id: backend-common-integrations-contact-details-from-scraped-pages
+domain: backend
+category: integrations
+applies_to: [general, firecrawl]
+confidence: verified
+sources:
+  - https://docs.firecrawl.dev/api-reference/endpoint/scrape
+last_verified: 2026-09-06
+related: [backend-common-integrations-robots-txt-and-source-selection, backend-common-reliability-timeouts-and-retries]
+---
+
+# Collecting Phone Numbers and Addresses From Company Sites With a Main-Content Scraper
+
+## When this applies
+
+Extracting contact details (phone, address, messenger handle) from company
+websites through a scraping API or library that offers a "main content only"
+mode (Firecrawl `onlyMainContent`, readability-style extractors); such a run
+returns clean text with few or no phone numbers; deciding whether to add a
+JavaScript-rendering fallback.
+
+## Do this
+
+1. **Turn the main-content filter off for contact extraction.** Sites put
+   contact details in the header bar, the footer, and floating widgets —
+   exactly the regions a main-content filter is defined to remove. Firecrawl's
+   `onlyMainContent` defaults to `true` and "excludes headers, navs, footers";
+   set it to `false` (or list `header`, `footer`, `aside` and the theme's
+   wrapper selectors in `includeTags`) so the extractor sees the regions where
+   the numbers live.
+2. **Extract from the raw HTML's link targets before any text heuristic.**
+   Request the HTML format alongside markdown and read `href` values:
+
+| Link | Yields |
+|------|--------|
+| `tel:+84…` | The canonical phone number, already normalized by the site |
+| `https://zalo.me/…`, `https://wa.me/…`, `viber://…` | Messenger handles that are the primary contact on many SME sites |
+| `mailto:` | Email without a regex over rendered text |
+
+   Fall back to a regex over the full-page text only for sites with no such
+   links.
+3. **Treat JavaScript rendering as the last fallback.** Measure before adding
+   it: fetch the static HTML and the rendered DOM for a sample of target sites
+   and diff the numbers found. Add rendering only for the sites where the diff
+   is non-empty.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| The number appears only as an image or CSS-obfuscated spans | Neither mode returns it as text; record the site as "contact by image" and stop, rather than adding OCR to the pipeline for one site |
+| The footer holds several numbers (branches, departments) | Keep all, labeled by the nearest heading or `aria-label`; a "first number wins" rule picks the fax line |
+| `includeTags` is used instead of `onlyMainContent: false` | Include the wrapper elements the theme actually uses (`#footer`, `.topbar`) — a site that renders its footer as a `<div>` is not matched by the `footer` tag alone |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Scrape in the default main-content mode and regex the markdown for phone numbers | Disable the main-content filter and read `tel:`/messenger hrefs from the HTML first | The default mode removes the header and footer where the numbers sit |
+| Add headless rendering because the static fetch found no number | Diff static vs rendered on a sample first | In the measured sample no site injected a number by JavaScript alone; the missing numbers were in the filtered-out regions |
+
+## Sources
+
+- https://docs.firecrawl.dev/api-reference/endpoint/scrape — `onlyMainContent` (default `true`): "Only return the main content of the page excluding headers, navs, footers, etc. This is a deterministic HTML-level filter applied before markdown is generated"; `includeTags` / `excludeTags`: "Tags to include in the output" / "Tags to exclude from the output"
+- Field measurement 2026-09-04 (30 Vietnamese company websites, static HTML vs rendered DOM): 16/30 sites carried their phone number only outside the main content (header, footer, floating widget); 22/30 exposed it as a `tel:` link; 0/30 gained a number after JavaScript rendering that the static HTML lacked
@@ -35,6 +35,7 @@ description or trigger line that decides when an instruction loads.
 | Output compliant in letter but wrong in shape | A positive recipe or template of the wanted shape — in head-to-head wording tests, adding prohibitions to a wrong-shape failure produced MORE of the unwanted content than no guidance |
 | A required element omitted | A structural template with the element as a REQUIRED slot |
 | Behavior that depends on the situation | Predicate-keyed rules ("when X, do A; when Y, do B") — a blanket rule with exemption clauses leaves the boundary to per-run improvisation |
+| The instruction said "ask the user" and the agent asked in prose (a numbered list typed into the reply instead of the question tool's chooser) | Name the channel, not only the shape — "ask with the `AskUserQuestion` tool" — and gate the action that depends on the answer with a PreToolUse hook that reads `transcript_path` and denies (exit 2) until a call to that tool exists in the transcript. Shape words ("numbered list", "one at a time") leave the tool choice open, and prose is the cheapest rendering; wording alone does not hold the choice across sessions, a hook checks it mechanically |
 
 3. Close loopholes from actual transcripts: quote the rationalization the
    agent produced and counter it by name; add counters only for observed
@@ -62,6 +63,7 @@ description or trigger line that decides when an instruction loads.
 | The instruction must hold across models or versions | Re-run the baseline+instruction pair per model — binding wording is model-specific, and an upgrade can un-bind it |
 | Two instruction sources conflict (a skill vs a project rule) | State the precedence inside the artifact itself; unstated precedence gets resolved differently per run |
 | The instruction is a one-off for a single session | Skip the test harness and state the rule directly — the testing cost is justified by reuse, and a session instruction is consumed once |
+| The gated question runs headless (no TTY) | The question tool resolves at once with empty answers, so the transcript carries a call with no human answer; have the hook also require a non-empty answer in the tool result, or route the decision out of band per [infrastructure-agent-orchestration-unattended-worker-questions] |
 
 ## Instead of
 
@@ -70,8 +72,11 @@ description or trigger line that decides when an instruction loads.
 | Stack a third prohibition after two were ignored | Rewrite as a recipe or predicate rules, then retest | Prohibition stacking measured worse than no guidance for wrong-shape failures |
 | Ship a wording after one clean test run | Run 5+ repetitions plus a no-guidance control | A single run samples the good tail of the distribution; the control proves the instruction is doing the work |
 | Write "never do X" on its own | Pair it with the replacement action that makes it unnecessary | A bare prohibition invites improvisation at the boundary — the same rule this wiki's AGENTS.md enforces on its own pages |
+| Fix a "asked in prose" regression by adding "use a numbered list" to the wording | Name the question tool in the wording and add the transcript-reading PreToolUse gate | The list instruction was already present when the regression happened; only the tool name plus a mechanical check binds the channel |
 
 ## Sources
 
 - https://github.com/obra/superpowers — writing-skills skill and its testing references: baseline-fails-first discipline, form-matched-to-failure table with head-to-head wording measurements, nuance-clause degradation, description-as-trigger-only rule; field-tested across the framework's own skill suite
 - https://github.com/ayghri/i-have-adhd/blob/main/skills/i-have-adhd/SKILL.md — "Pre-send check": each output rule paired with a pre-emit self-check predicate and its fix
+- https://code.claude.com/docs/en/hooks — PreToolUse hook input carries `transcript_path` (the session's JSONL transcript) and `tool_input`; exit code 2 blocks the tool call and shows stderr to the model
+- https://github.com/choiyounggi/dev-loop/pull/144 with `hooks/orchestrate-ask-gate.sh` (https://github.com/choiyounggi/dev-loop/blob/main/hooks/orchestrate-ask-gate.sh) — field reproduction 2026-08-25: PR #136 introduced the gate wording ("ask in one numbered round") without naming the tool and Gate 1 degraded to prose questions; #144 named `AskUserQuestion` in the skill text and added a PreToolUse(Bash) gate that scans the transcript for that tool call before the first worker launch (27 bats cases; 744 tests green)
@@ -95,7 +95,7 @@ Match your situation to a "load when" line; load only matching pages.
 |------|-----------|
 | [completion-response-validation](common/llm/completion-response-validation.md) | Consuming OpenAI-compatible `/chat/completions` output as a final artifact (summary, document, notification); LLM responses coming back empty or truncated while HTTP status is 200; a reasoning-family model may be routed onto the alias you call |
 | [context-window-budget](common/llm/context-window-budget.md) | Repointing an LLM client or agent CLI at a different model, a self-hosted server (vLLM/Ollama), or a gateway (LiteLLM); setting `max_tokens` for a client whose default was sized for a larger model; the first request after such a switch returns 400 with a context-window error; deciding where to set the cap (request body vs client env var vs gateway config) and how to point the base URL at a proxy; handling truncation that arrives as a normal 200 |
-| [binding-instructions-for-agents](common/llm/binding-instructions-for-agents.md) | Authoring or editing a skill, system prompt, hook message, or CLAUDE.md/AGENTS.md rule that must change an agent's behavior under pressure; an instruction keeps getting rationalized around; choosing between prohibition, recipe, template-slot, and predicate-rule forms; writing a skill's description/trigger line; adding a pre-send self-check to an output-shape rule |
+| [binding-instructions-for-agents](common/llm/binding-instructions-for-agents.md) | Authoring or editing a skill, system prompt, hook message, or CLAUDE.md/AGENTS.md rule that must change an agent's behavior under pressure; an instruction keeps getting rationalized around; choosing between prohibition, recipe, template-slot, and predicate-rule forms; writing a skill's description/trigger line; adding a pre-send self-check to an output-shape rule; an instruction said "ask the user" and the agent asked in prose instead of through the question tool |
 | [progressive-disclosure-artifacts](common/llm/progressive-disclosure-artifacts.md) | Authoring a skill or reference bundle that packages a large corpus (a book, manual, wiki) for an agent to consult on demand; an always-loaded skill file grew past a few thousand tokens; sizing per-chunk token budgets by content type and usage depth; an agent Reads an entire large source file to answer narrow questions |
 
 ### ml
@@ -112,6 +112,7 @@ Match your situation to a "load when" line; load only matching pages.
 | [consumer-required-fields](common/integrations/consumer-required-fields.md) | Writing an adapter that maps one module's records into the payload a second module (scoring engine, plugin, external client) consumes, with the target shape taken from a docstring, README example, or sample payload; such an adapter runs end to end with no error and the downstream numbers come out low; deciding which mapped fields need their own assertion |
 | [robots-txt-and-source-selection](common/integrations/robots-txt-and-source-selection.md) | Choosing which site to fetch a published dataset from and reading its robots.txt to decide whether your client may crawl it; the file contains a `Disallow: /` and you are deciding whose group it belongs to; setting the crawler's User-Agent and checking that token against the file; robots.txt returned a non-200 status; the origin restricts your token and you are looking for a portal that republishes the same records |
 | [estimate-derived-thresholds](common/integrations/estimate-derived-thresholds.md) | Submitting an action to an external system whose actual outcome can differ from the decision-time estimate (market order fill vs quote) while persisting absolute trigger values derived from that estimate (SL/TP prices, alert thresholds); derived triggers fire immediately or at the wrong level right after the action confirms; choosing where to recompute them from the actual outcome |
+| [contact-details-from-scraped-pages](common/integrations/contact-details-from-scraped-pages.md) | Extracting phone numbers, addresses, or messenger handles from company websites through a scraper with a "main content only" mode (Firecrawl `onlyMainContent`); the run returns clean text with few or no numbers; deciding whether to add a JavaScript-rendering fallback; choosing between `tel:`/messenger `href` extraction and text regex |
 
 ### storage
 

@@ -9,7 +9,7 @@ Match your situation to a "load when" line; load only matching pages.
 
 | Page | Load when |
 |------|-----------|
-| [reproduce-first](methodology/reproduce-first.md) | A bug is reported or behavior is wrong and you are about to investigate or fix; deciding what to capture when full reproduction is impossible (prod-only, timing-dependent, one-off crash) |
+| [reproduce-first](methodology/reproduce-first.md) | A bug is reported or behavior is wrong and you are about to investigate or fix; deciding what to capture when full reproduction is impossible (prod-only, timing-dependent, one-off crash); a script's command pasted into an interactive shell fails to reproduce the script's bug; a green test is about to be cited as proof a stage is healthy |
 | [isolate-by-bisection](methodology/isolate-by-bisection.md) | A bug reproduces but its location is unknown; it worked before / works in env A but not env B / fails with one input but not another — binary-searching versions (git bisect), code paths, data, or environment diffs |
 | [hypothesis-testing](methodology/hypothesis-testing.md) | You have a suspect cause and are about to "try a fix"; several suspects compete and you must pick what to test next; verifying that a fix that "worked" actually addressed the mechanism |
 | [probe-path-vs-operation-path](methodology/probe-path-vs-operation-path.md) | A precondition probe (login status, health, connectivity) reports success while the operation it gates fails with an auth/permission error; a browser page-load login check gates direct API calls made with stored cookies; deciding what a preflight probe must exercise under refresh-token cookie auth |