DIT-13480: Capture the raw literal (snapshot_text) in resource-file extractors

[laurakoye]

Overview

Fills in snapshot_text for the resource-file extractors and makes the field required.

snapshot_text is the contiguous source region a value came from — not "the region it's safe to write to." Every case has one, so the field is never null. value_raw is decoded and can't be searched for; the snapshot is a verbatim slice, so source.includes(snapshot_text) always holds.

Most formats were already computing the span and discarding it — json-i18n and xcstrings tracked start/end but read only start, strings had endIndex, arb inherits from json-i18n, and the CDATA paths captured their payload verbatim. The rest:

Format	Span
XML (android-resources, resx, xliff)	the element's inner text, nested tags and entities included
yaml-i18n	block and folded scalars include the | header and per-line indentation
.po	first quote of a chunk group through the last quote of its final line
.properties	the value's first character through the end of the last backslash-continued line

.po and .properties read from the original source via a shared line-offset table rather than re-joining split lines, so CRLF files round-trip. .properties maps indices in the assembled logical line back to their physical line, so a bundle that continues the key (long\ / key = Value here) spans only the value.

The type flip is what makes it stick — the compiler now catches any extractor that forgets:

File	From	To
src/lang/types.ts	snapshotText?: string	snapshotText: string
src/extract.ts	hit.snapshotText ?? null	hit.snapshotText
src/types.ts	z.string().nullable()	z.string()

Editability is a separate, write-path concern: a writer must refuse to replace a span carrying markup or syntax the value doesn't. Nothing writes back yet, so that's a gate on DIT-13471, tracked in DIT-13481.

Also fixed: .stringsdict truncated its values

elementText returned only the first text child, and the HTML grammar splits text at every entity reference and nested tag:

"Tom & Jerry"  → value "Tom"
"Hi <b>there</b>"  → value "Hi"
"50 &#37; off"     → value "50"

Keys truncated the same way, producing a wrong i18nKey — or failing the USER_FACING_KEYS lookup and dropping the entry. stringsdict was the only XML format not using xml.ts's innerText, which reads the whole inner region, strips nested tags, and decodes entities; it now does, and elementText is gone. Note entities now decode in the value (Tom & Jerry), matching the other three XML formats, and snapshotText is the raw span rather than a copy of the decoded value.

Also fixed: a po hit's location pointed at the wrong line

A hit's location came from the msgid side even when its value came from a msgstr[N], so location and snapshot_text described different places — line 6 for text living on line 8. It now comes from whichever chunk supplied the value, so the two agree:

msgid "unread_one"            line 6
msgid_plural "unread_many"    line 7
msgstr[0] "1 unread message"  line 8   → location 8, snapshot "1 unread message"
msgstr[1] "%d unread ..."     line 9   → location 9, snapshot "%d unread messages"

pickNonEmpty already returned the winning chunk, which carries its own line, so this is a two-word change plus tests asserting each snapshot is findable on the line its location names.

Flagged for review, not changed

• Duplicate candidate ids, pre-existing. makeCandidateId hashes (file, line, column, value) — the line, not the occurrence index. PO plurals with three or more forms share a location line, so two forms with identical text collide. occurrence_index distinguishes them; the id doesn't. Tracked in DIT-13503 — the location fix above doesn't resolve it, since the colliding forms are still on one line.

• XLIFF <source> text is dropped when it carries an inline placeholder. <source> is a void element in HTML, and this project parses XML with ast-grep's HTML grammar — so when a <g>/<ph>/<xliff:g> appears inside it, the grammar closes <source> early, makes the placeholder a sibling, and demotes </source> to erroneous_end_tag. innerText requires an end_tag, finds none, and returns null, so no candidate is produced. <target> is unaffected — it isn't an HTML element, so it nests normally:

  <source>Hello <g id="n">%s</g></source>   → nothing
  <target>Bonjour <g id="n">%s</g></target> → "Bonjour %s", snapshot "Bonjour <g id=\"n\">%s</g>"
  

  So a unit keeps its translation and silently loses its original, and a base-locale XLIFF loses the copy that matters. Pre-existing, documented by a test, tracked separately.

• YAML aliases are dropped. b: *x parses to an Alias node with no value property, so the walker's isScalar guard skips it. Unrelated to the XLIFF cause despite the similar symptom — tracked separately.

Context

DIT-13480, part of DIT-13405. Based on main.

A code link identifies a string by (file, value, occurrence index) and stores snapshot_text as the code's own rendering of it. Schema design.

ZCodeLink.snapshotText is non-nullable and required: true in mongoose (ditto-app#9346), so a null snapshot couldn't be persisted as a codelink at all — the field being always-present is what makes rung 2 of reconciliation (snapshot_text + occurrence_index) usable when an i18n key is renamed.

Screenshots

None — extraction output only.

Test Plan

1. The suite

yarn jest        # 23 suites, 197 tests — 37 new
yarn typecheck   # clean

Eleven extractors joined the describe.each table in snapshot-text.test.ts, so every format now asserts snapshotText is present on each hit and findable verbatim in the source. A second describe pins the spans that aren't replaceable as-is: XML wrapping a nested child, YAML block scalar beside a plain sibling, PO multi-chunk, .properties continuation, and a continued key.

A new sweep covers the four XML formats: every span must run from one tag boundary to the next (> … <, or the CDATA brackets). source.includes() proves a span is findable, never complete — "Tom" satisfies it trivially — which is why the truncation above survived the other checks. Verified by reintroducing the bug: the sweep fails on stringsdict and passes on the other three.

strings.test.ts also needed updating, since it asserts whole hit objects with toEqual.

CI (.github/workflows/ci.yml) runs typecheck, jest, and build on Node 20 and 22 for every PR.

2. Coverage on a real app

yarn build

mkdir -p /tmp/demo
git -C ../v0-demo archive HEAD | tar -x -C /tmp/demo

cat > snap.mjs <<'EOF'
import { runExtract } from "./dist/index.js";
import { readFileSync } from "node:fs";
import { join } from "node:path";
const root = process.argv[2];
const { candidates } = await runExtract({ inputPath: root });
const missing = candidates.filter((c) => !c.snapshot_text);
console.log(`${candidates.length - missing.length}/${candidates.length} carry a snapshot`);
console.log("missing kinds:", [...new Set(missing.map((c) => c.detection_kind))]);
const cache = new Map();
const bad = candidates.filter((c) => {
  const f = c.location.file;
  if (!cache.has(f)) cache.set(f, readFileSync(join(root, f), "utf8"));
  return !cache.get(f).includes(c.snapshot_text);
});
console.log(`spans found verbatim in their file: ${candidates.length - bad.length}/${candidates.length}`);
EOF

node snap.mjs /tmp/demo
rm snap.mjs

snap.mjs goes in the repo root — ./dist/index.js resolves relative to the script, not the working directory.

3. What to look for

200/200 carry a snapshot
missing kinds: []
spans found verbatim in their file: 200/200

The second and third lines are the properties that matter: nothing is missing a span, and every span is a real slice of its own file. v0-demo's resource files are JSON catalogs, so the stitched and multi-line paths are covered by unit tests instead.

Note: yarn linking this into the CLI and running yarn scan does not exercise this code until cli#151 merges — the CLI still uses its own copy.

[jholiga]

Testing looked good! Couple smaller things I noted, one is a unit test failure, and the other is something I found with Claude while testing, that appears to be a pre-existing extractor bug but might be more exposed with this change being introd

[jholiga]

I think you noted this in the description but these tests seem to be failing

[jholiga]

Couple small things I saw, but I think they're minor! Both relate to location.column in different extractors