P5-crossref: animedex crossref (vendored nattadasu/animeApi snapshot)

[narugo1992]

Goal

Land Phase 5's cross-source ID translation: animedex crossref <prefix:id> [--from <backend>] [--deep]. The command takes a prefix-encoded ID (the same shape P5-search-show introduces for show) and returns the equivalent IDs across every other catalogue the project knows about. It runs offline against a vendored snapshot of nattadasu/animeApi committed to the repo as animedex/data/crossref.json; the snapshot only covers anime today. The --deep flag is a documented but deferred escape hatch that points at Phase 7 (AniDB), and raises an informative error until that track lands.

Refs #1 §7 (Phase 5 checklist).

Why this slice

crossref is the supporting primitive for Phase 5 multi-source merging: season and search already merge AniList / Jikan (and friends) rows that identify the same anime, using shared external IDs first and a deterministic fuzzy comparison as a fallback. crossref makes the "shared external IDs" half of that contract independently inspectable and reusable — given an anilist:154587, what are the equivalent IDs on every other catalogue. The "this anime is also on anidb" footers on show, the cross-source ID translation any agent might want to do mid-pipeline, and any future tightening of the merge matcher all consume this primitive. This PR is the right place to ship it because by the time P5-search-show has merged, the prefix:id parser exists; this PR's only new substrate is the snapshot-loading module.

It is also the lightest of the three Phase 5 slices (one command, no fan-out, no HTTP at runtime for the typical path), which makes it the natural pickup for whichever codex finishes its prior slice first.

Dependencies

This issue depends on P5-search-show being merged because it reuses the prefix:id parser at animedex/agg/_prefix_id.py. Do not start implementation before that PR lands; the parser shape is the foundation. Start writing the abstraction proposal (next section) once P5-search-show's proposal has been signed off — that gives you the locked parser contract to design against.

Scope (minimum required)

animedex crossref <prefix:id> [--from <backend>] [--deep]

• <prefix:id> follows the canonical shape from P5-search-show: anilist:154587, mal:52991, kitsu:43534, shikimori:52991, mangadex:<uuid>, anidb:<numeric>.
• --from <backend> is an optional disambiguator. Use only when the input is a bare numeric ID without a prefix — i.e. when the caller piped a raw ID from somewhere else. If both prefix and --from are present, they must agree, otherwise the command exits with a clear "prefix backend disagrees with --from" error.
• --deep opt-in flag that promises to also do a live AniDB roundtrip for any input ID. Until Phase 7 lands, --deep raises an ApiError(reason="...") with a message that explains the feature is not yet shipped and points the user at the raw passthrough animedex api anidb /... (which will also not work until Phase 7's transport bits land — that's accurate; the user knows where to look).
• The output schema is a structured envelope listing every known equivalent ID:

{
  "_input": {"backend": "anilist", "id": "154587"},
  "matches": [
    {"backend": "anilist", "id": "154587", "_self": true},
    {"backend": "mal",     "id": "52991"},
    {"backend": "anidb",   "id": "17821"},
    {"backend": "kitsu",   "id": "43534"},
    {"backend": "themoviedb", "id": "209867"}
  ],
  "source": {
    "kind": "vendored-snapshot",
    "name": "nattadasu/animeApi",
    "snapshot_date": "2026-05-09T00:00:00Z",
    "note": "anime entity only; manga / character / person crossref not supported by the vendored map"
  }
}

(Field names are author's choice; settle the exact schema in the proposal step. The _self marker on the input row is one option; another is to always emit it as the first entry and not annotate.)

• Misses are not errors. If the input ID has no entries in the vendored map, return {"_input": {...}, "matches": [], "source": {...}} with an informative TTY message and stderr line; the JSON path stays structurally valid for jq consumers.

• Entity scope: anime only. The vendored snapshot does not cover manga / character / person / studio. If the prefix names a non-anime backend that the project supports (e.g. mangadex:<uuid>), the command exits with a clear "crossref currently only covers anime; mangadex IDs are manga and have no cross-source map yet" error message rather than silently returning zero matches.

The vendored map: animedex/data/crossref.json

• Source: nattadasu/animeApi repository's release artifact (typically named animeapi.json or similar). Inspect the repo's release tab for the canonical filename and a stable raw URL. As of writing, the maintained mirror at https://github.com/nattadasu/animeApi/raw/main/database/animeapi.json is the recommended source; verify before snapshotting.
• Snapshot mechanics: vendored into the repo at animedex/data/crossref.json via a one-shot capture script (suggest tools/data/snapshot_crossref.py or extend an existing capture tool). The commit message that lands the snapshot states the capture date and the source URL.
• Snapshot shape: the file is a list of records, each record carrying every known ID for one anime title. The vendored file is consumed via importlib.resources exactly like animedex/data/ghibli.json.
• A _snapshot_date (or equivalent) field at the top level is added by the snapshot tool so the runtime can surface the snapshot age in the source envelope above without needing to look at file mtime.
• File size: expect 5–15 MB depending on the upstream's coverage. PyInstaller smoke must still pass; add the data file to HIDDEN_DATAS or the spec's datas= list if it does not already pick up data from animedex/data/.

Lifecycle note (for the docstring and PR body)

• The snapshot ages. Anime added to AniList in Aug 2026 will not be in a 2026-05 snapshot. The docstring of the crossref command and the _BACKEND_POLICY guidance line both state this fact plainly and recommend re-snapshotting at each minor release.
• The source.snapshot_date field on every response makes the staleness visible to the caller / agent.
• Do not add a "snapshot is older than X days" warning — that is over-protection. The user reads the date if they want.

Encouraged exploration

The minimum scope above is the floor. Sensible extensions:

• A tools/data/refresh_crossref.py script (committed but not auto-run) that fetches the latest upstream and updates animedex/data/crossref.json. Maintainers run this manually when they want to re-snapshot.
• --all / --include-self toggles to control whether the input ID is echoed in matches.
• A animedex crossref --list-backends subcommand or output that names every backend the vendored map can translate to (anilist, mal, kitsu, anidb, anime-planet, themoviedb, etc.). Useful for agents discovering capability.
• A bulk variant: cat ids.txt | animedex crossref --stdin reading prefix:id per line, emitting JSON lines.

Each lands only if it fits without inflating the PR. If you add any, surface it in the PR body.

API documentation entry points

• The vendored upstream: https://github.com/nattadasu/animeApi. Read the README to confirm the current canonical artifact filename and license stance (the project is permissive; verify and reflect in the snapshot script's commit message).
• The prefix:id parser: animedex/agg/_prefix_id.py — landed by P5-search-show.
• The animedex/data/ consumption pattern: animedex/backends/ghibli/__init__.py:_load_snapshot() for the canonical importlib.resources + frozen-binary-fallback shape.

Substrate touch points (read carefully)

This PR's substrate footprint is small (one data file, one loader, one command), but the choices around the loader and the deferred-feature error are worth a brief proposal. Before writing implementation code, post an abstraction proposal as a comment on this issue and wait for an explicit external sign-off — sign-off is a recorded maintainer reply, not a self-reply (AGENTS §15.5; merged PR #14 and PR #15 issue comments show the well-formed shape).

The proposal should answer at least:

1. Loader shape. Two natural options:

   • (a) Load the JSON eagerly at module import time into a top-level dict keyed by (backend, id). Constant lookup. ~5–15 MB resident.
   • (b) Lazy-load on first call. Same memory footprint after first use; cold start cheaper.
   • For a CLI that may run as a one-shot subprocess, (b) is friendlier; for an MCP server long-running, (a) is friendlier. Propose one with rationale; both are acceptable.

2. Internal index shape. The vendored JSON is a list-of-records. Building a (backend, id) → record dict gives O(1) lookup. Building a list of records + linear scan gives O(N) per query. Pick (a) given expected query volume; the build cost is paid once per process. Confirm.

3. --deep deferred-feature error. The ApiError.reason enum already supports "auth-required" and several other reasons. For "this feature exists in spec but is not yet shipped", we can (a) reuse "auth-required" (semantically wrong but pre-existing), (b) add a new reason like "deferred-feature" or "not-implemented" to animedex/models/common.py:REASONS. Adding a new reason is the cleaner shape; propose the exact string. The error message text explicitly avoids the phrase "Phase 7" (§14) and instead uses product-surface wording: "deep crossref requires an AniDB high-level helper, which is not yet shipped; track the project's roadmap or use animedex api anidb /... once the raw passthrough is wired".

4. --from disambiguator semantics. When both prefix and --from are present and disagree (e.g. crossref anilist:154587 --from mal), exit with a clear error rather than silently honouring one. Settle the precise wording.

5. Manga / non-anime prefix handling. When crossref mangadex:<uuid> is invoked, exit with the entity-scope error described above rather than returning empty matches. Confirm the wording is informative and names the entity types currently supported.

6. PyInstaller bundling. Confirm animedex/data/crossref.json is picked up by the existing spec / hidden-datas list. If not, add it. Frozen smoke must pass.

If the proposal you write diverges materially from the above, that is fine; the proposal exists to surface the trade-off in writing before code lands.

Fixture capture / snapshot capture

• The snapshot itself is committed verbatim as animedex/data/crossref.json. The capture tool that produces it lives at tools/data/snapshot_crossref.py (or similar) and is run once per re-snapshot operation, manually.
• Test fixtures are minimal: a handful of sample crossref translations exercised against the real (committed) crossref.json. Tests do not need HTTP mocks because the runtime is offline; responses.RequestsMock is therefore not used. Tests do need to assert against representative records picked from the snapshot — pick a few well-known anime IDs (Frieren / Bleach / Naruto / Cowboy Bebop) as targets.
• --deep fixture: a unit test that calls the command with --deep and asserts the typed ApiError is raised with the agreed message text. No HTTP needed.
• Document the snapshot date in the PR body and the snapshot script's commit message.

Verification checklist (self-check before requesting review)

• animedex/data/crossref.json exists and contains the vendored snapshot with a _snapshot_date head field.
• tools/data/snapshot_crossref.py (or named equivalent) is committed and re-runnable; running it produces a byte-identical crossref.json against the same upstream content (idempotence).
• animedex crossref anilist:154587 (Frieren) returns a non-empty matches list including mal, anidb, kitsu, and at least one external (themoviedb / anime-planet) entry.
• animedex crossref mal:52991 resolves to anilist:154587 plus the other equivalents — round-trips with the above.
• animedex crossref kitsu:43534 resolves similarly.
• animedex crossref anidb:17821 resolves similarly without requiring --deep (the vendored map already contains AniDB IDs for known anime; --deep is reserved for live AniDB lookups not present in the map).
• animedex crossref anilist:9999999999 (valid format, missing from the map) returns matches: [] with a stderr inform line; exits 0.
• animedex crossref badprefix:1 exits with a clear "unknown prefix" error message naming the supported prefixes.
• animedex crossref anilist:abc exits with a clear "ID is not numeric" error before any lookup.
• animedex crossref anilist:154587 --from mal exits with the disagreement error.
• animedex crossref mangadex:<some-uuid> exits with the entity-scope error (manga not covered).
• animedex crossref anilist:154587 --deep raises the typed ApiError with the agreed deferred-feature message text.
• CLI tested in both --json (forced JSON) and the default TTY path with isatty()=True forced (AGENTS §9bis.6).
• _BACKEND_POLICY entry under crossref (if registered as a top-level Click group) with backend_line, rate_line, guidance strings. rate_line reads something like "Not applicable; the high-level crossref command serves from the bundled offline snapshot."
• _SELFTEST_TARGETS registers animedex.agg.crossref (and the loader module if separated).
• The loader's selftest() reads the vendored JSON, asserts top-level shape (list with _snapshot_date), asserts at least one well-known ID translates correctly. Offline. Fast.
• Tutorial entry at docs/source/tutorials/crossref.rst with at least one runnable example for each of (a) anime input, (b) miss, (c) --deep deferred error.
• make rst_auto regenerates docs/source/api_doc/agg/ and the diff is committed.
• make build && make test_cli passes — the frozen binary correctly bundles the JSON.
• Pre-flight (AGENTS §15.4): full sequence green; grep -rE 'Phase [0-9]|AGENTS[. ]§|Reviewer review' animedex/ tools/ returns zero.

Load-bearing reminders

• Snapshot staleness is the user's call. The source.snapshot_date field surfaces the date; the docstring states the lifecycle expectation; the project does not add a "snapshot too old" warning. §0 inform-not-gate applies: inform the date, do not block on it.
• Deferred-feature error wording avoids §14 traps. Do not write "Phase 7 not implemented" in the error text; use product-surface wording like "AniDB high-level helper not yet shipped". The phrase "phase" inside animedex/ / tools/ is forbidden by §14 and the grep step in pre-flight catches it.
• The vendored JSON is consumed via importlib.resources, matching the ghibli.json pattern (animedex/backends/ghibli/__init__.py:_load_snapshot). Do not re-invent a path resolver; the frozen-binary fallback in that helper is the canonical shape.
• HTTP-only test seam for any HTTP code paths. This command does no HTTP at runtime, so the rule applies vacuously — but the --deep future implementation will use responses.RequestsMock and not monkeypatch.setattr.
• §15.2 read-only coverage — the command exposes every prefix the vendored map can resolve. If the map contains backend IDs the project doesn't otherwise know about (e.g. anime-planet, notify-moe), surface them in the output even though no animedex command consumes them yet; an agent's downstream pipeline may.

Parallelism

This issue is serial after P5-search-show. Start the abstraction proposal as soon as P5-search-show's proposal has been signed off (so the parser contract is locked); start implementation when P5-search-show merges.

It is independent of P5-calendar — calendar's fan-out helper is a different code path and they do not share any source file beyond the common animedex/agg/ namespace.

PR body template (what to include)

When you open the PR, please include:

• Summary — one-paragraph overview of the command, the vendored snapshot, the deferred-feature error for --deep.
• Demo — TTY GIF for animedex crossref anilist:154587 (Frieren), ideally showing both the happy match list and the deferred-feature --deep error.
• Examples and expected output — copy-pastable command lines including the miss case and the entity-scope-error case.
• Snapshot notes — capture date, source URL, snapshot size (rough), the upstream's license (and our compliance posture if needed; reuse the same posture as ghibli.json).
• Fixture notes — none from HTTP (offline runtime); list the sample IDs used in tests.
• Verification — tick the checklist; mark each as done / partial / deferred with a one-line reason.
• Abstraction proposal link — link to the comment on this issue where you settled the loader / error-reason design, plus the maintainer reply that signed off (mandatory per §15.5).

Out of scope

• Live AniDB roundtrip on --deep (Phase 7).
• Manga / character / person crossref (vendored map does not cover; future work depends on a different source).
• Automated re-snapshot on a schedule or in CI (manual maintainer action only).
• Tightening the existing merge matcher in season / search to consult crossref as a stronger external-ID signal (future refinement; this PR ships the primitive, and consumers can adopt it independently).
• Validation that a translated ID is live on its target backend (no probe; offline runtime).
• A reverse "show me every anime that has only N source coverage" report (analytics; out of scope).

[narugo1992]

Unblocked: PR #22 has merged (commit 138281ef). animedex/agg/_prefix_id.py is now on main — this issue's <prefix:id> parsing dependency is resolved. Safe to start the implementation slice.