Skip to content

Master implementation plan: Phase 0 → 8 (full v1 + AniDB) #1

Description

@narugo1992

This issue is the master tracking document for the full implementation of animedex from the current scaffold to a v1.0 release that covers every phase in plans/04-roadmap-and-mvp.md, including the heavy AniDB track. Sub-issues and PRs should reference this issue as Refs #N. The checklists below are the source of truth for "what still needs to land".

The design rationale is fixed in plans/01..05 and AGENTS.md; this issue does not re-litigate it. It only enumerates the executable checklist.


0. Tech stack snapshot

0.1 Locked-in (do not re-discuss)

Decision Choice Source
Project shape read-only multi-source CLI for anime/manga plan 03
Command flavor gh-flavored (backend verb + api passthrough) plan 03
Policy carrier docstring (Backend: / Rate limit: / Agent Guidance triplet) plan 02
Source attribution mandatory _source / [src: ...] annotation everywhere plan 03 §5
CLI framework Click 8 requirements.txt
Sync HTTP requests 2.28+ requirements.txt
Packaging PyInstaller 6 (spec-based) + sdist + wheel landed
Style ruff (120-col) + flake8 + reST docstrings landed
Docs Sphinx + RTD theme + auto_rst pipeline landed
Token storage OS keyring plan 02 P1
Repo language English-only repo content AGENTS §1
Identity model per-repo git config + per-invocation GH_TOKEN AGENTS §2-3

0.2 Decided this round

Decision Choice Side effect
Models layer pydantic v2 BaseModel adds runtime dep; PyInstaller bundle grows ~4-5 MB; cache uses model_dump_json() directly
Python floor 3.9+ (drop 3.7 and 3.8) update setup.py, python_requires, classifiers, CI matrix

0.3 Still open (default answers given; resolve before they bite)

#3 corrects several plan-01 facts that have drifted by 2026-05-07 (kitsu.app/.io parity, shikimori.one still serving, MangaDex Via no longer enforced, Trace.moe anonymous quota dropped from 1000 to 100/month, no rate-limit headers exposed by Jikan/MangaDex/Danbooru). Read #3's drift register before touching plan 01.

  • MCP registration timing: eager (import-time side effect) vs lazy. Default applied (PR Phase 0: substrate (models / transport / cache / auth / render / policy / mcp) #2): lazy via explicit register_animedex_tools(server) entry point — animedex/mcp/register.py has no import-time side effects, confirmed by its selftest().
  • animedex.aio async layer launch: Default applied: deferred until after v1.0. All shipped code (Phase 0+1+2 + jq wheel) is sync; no animedex.aio module exists in the tree.
  • MyAnimeList official API v2 backend fate: read-only constraint makes Jikan a full replacement. Default applied: explicitly dropped — the package ships no MAL backend, Jikan covers every MAL anonymous read.
  • AnimeChan filtered quote access: Default updated by PR Add Ghibli and quote backends #14: the high-level AnimeChan v1 surface exposes random, random-by-anime, random-by-character, quotes-by-anime, quotes-by-character, and anime; the older premium-only flag framing is no longer the project shape.

1. Roadmap (bird's-eye)

Phase 0   substrate            4-5 d   ─── critical path; everything depends on it
Phase 1   api passthrough      2 d         ─── 8 backends, validates substrate
Phase 2   MVP commands         2-3 d           ─── anilist/jikan/trace/nekos
Phase 3   mid-tier read        3 d                 ─── kitsu/mangadex(no reader)/danbooru/waifu
Phase 4   cal/news/trivia      2 d                       ─── shikimori/ann/ghibli/quote
Phase 5   aggregate            2 d                             ─── search/show/crossref/season/schedule
Phase 6   mangadex reader      1.5 d                                 ─── pages subcommand

Phase 7   AniDB heavy track    5-8 d   ─── independent track; can run in parallel with 1-6
Phase 8   polish               2 d         ─── completion/alias/--web/MCP server/docs build

Release cadence:

  • 0.1.x = MVP (Phase 0+1+2 done)
  • 0.2.x = mid-tier + aggregate (Phase 3+4+5)
  • 0.3.x = mangadex reader (Phase 6)
  • 0.x.x+anidb build metadata = AniDB landing (Phase 7)
  • 1.0.0 = docstring lint green + Linux/macOS smoke matrix green + JSON schema stable

2. Phase 0 — Substrate (4-5 d) [the highest-leverage phase]

Each sub-piece is written once and reused by every subsequent phase. Cutting corners here costs us in every later phase.

2.1 Stack-level prerequisites

  • Bump python_requires to >=3.9 in setup.py; drop 3.7/3.8 classifiers.
  • Add pydantic>=2,<3 to requirements.txt.
  • Add platformdirs>=3 to requirements.txt (cache + token-store paths).
  • Add keyring>=24 to requirements.txt.
  • Update CI matrix to remove 3.7/3.8 and add 3.9..3.13.

2.2 animedex/models/ (pydantic v2 BaseModel layer)

  • models/common.py: SourceTag(backend, fetched_at, cached, rate_limited), Pagination, RateLimit, ApiError.
  • models/anime.py: AnimeTitle, AnimeRating, AnimeStreamingLink, Anime(source: SourceTag, ids: dict[str, str], ...).
  • models/manga.py: Manga, Chapter, AtHomeServer.
  • models/character.py: Character, Staff, Studio.
  • Shared AnimedexModel(BaseModel) base with populate_by_name=True, extra='ignore', frozen=True.
  • selftest() instantiates each model from a fixture and verifies schema.

2.3 animedex/transport/ (HTTP client + rate limiting + UA)

  • transport/http.py: HttpClient wrapping requests.Session per backend.
  • transport/ratelimit.py: in-memory token-bucket; one bucket per backend.
  • transport/useragent.py: single source of UA string with version + contact email.
  • transport/read_only.py: advisory known-read classifier for cache eligibility; raw passthrough no longer rejects caller-supplied methods. (Updated by PR Add raw API universal flags #17.)
  • selftest() exercises the advisory known-read classifier + UA injection with mocks. (Updated by PR Add raw API universal flags #17.)

2.4 animedex/cache/ (SQLite TTL cache)

  • cache/sqlite.py: keyed by (backend, request_signature); rows store (response_bytes, fetched_at, ttl).
  • Default TTL table: metadata 72 h, list 24 h, schedule 1 h, offline dump 30 d.
  • Path resolution via platformdirs.user_cache_dir("animedex").
  • pydantic models serialize via model_dump_json(); loads via Model.model_validate_json().
  • selftest() opens a tmp DB, writes one row, reads it back, expires it.

2.5 animedex/auth/ (OS keyring token store)

  • auth/keyring_store.py: real keyring backend, namespaced per backend.
  • auth/inmemory_store.py: in-memory backend for tests and headless CI.
  • Config.token_store plug point (interface defined in 2.8).
  • CLI commands: animedex auth status / auth login <backend> / auth logout / auth token.
  • selftest() does NOT touch the real keyring; only verifies the in-memory backend round-trip.

2.6 animedex/render/ (source-attributed renderer)

  • render/tty.py: human-friendly table renderer with [src: ...] suffixes.
  • render/json_renderer.py: full schema with per-field _source.
  • render/field_projection.py: --json field1,field2,... projection.
  • render/jq_filter.py: --jq <expr> via jq subprocess; gracefully degrades when jq is absent.
  • render/template.py: --template <jinja2> (optional).
  • render/attrib_off.py: --source-attribution=off (JSON only; TTY always shows source).
  • TTY-vs-pipe auto switch via sys.stdout.isatty().
  • selftest() round-trips a fixture Anime instance through both default renderers.

2.7 animedex/policy/ (docstring lint + --agent-guide)

  • policy/lint.py: walks animedex/backends/** and animedex.api.call, asserts every @cli.command(...) and @mcp.tool(...) docstring contains Backend:, Rate limit:, --- LLM Agent Guidance ---, --- End ---.
  • Wire into make lint.
  • animedex --agent-guide CLI: concatenates all Agent Guidance blocks across registered commands.
  • selftest() runs lint with no backends (Phase 0 state); confirms the runner returns 0.

2.8 animedex/config/profile.py (programmatic flag stack)

  • Config(BaseModel) with one field per CLI flag (cache_ttl_seconds, no_cache, rate, source_attribution, user_agent, timeout_seconds, token_store).
  • Config() (no args) reproduces unflagged CLI behavior.
  • Each public API function accepts config: Config | None = None.
  • selftest() instantiates default and fully-populated Config.

2.9 animedex/mcp/ (MCP server scaffolding, lazy)

  • mcp/register.py: register_animedex_tools(server) entry point, no import-time side effects.
  • mcp/tool_decorator.py: @mcp.tool lightweight wrapper that turns the docstring's Agent Guidance block into the MCP tool description.
  • MCP deps go in requirements-mcp.txt extras, not the runtime baseline.
  • animedex mcp serve CLI is stubbed in Phase 0; wired in Phase 8.
  • selftest() imports register.py and confirms it has no side effects.

2.10 Phase 0 gating criteria

  • make format && make test && make build && make test_cli all green.
  • python -m animedex.policy.lint animedex/ passes (vacuously, no backends yet).
  • animedex --agent-guide outputs an empty table.
  • animedex selftest reports every Phase 0 module under "smoke" not "import only".
  • A throwaway _example backend exercises the full substrate end-to-end (committed and removed in the same PR or kept as a template).

2.11 Phase 0 risks (carry forward)

Risk Likelihood Mitigation
pydantic-core fails to import in PyInstaller frozen binary medium early make build && make test_cli; list pydantic_core._pydantic_core in HIDDEN_IMPORTS
keyring has no backend on headless Linux CI high InMemoryStore + monkeypatch in tests; CI never touches the real keyring
--jq subprocess fails on Windows (no jq) high feature-detect; emit a friendly "jq not found" message; do not fail CI
pydantic frozen=True conflicts with backend-side scratch construction medium keep mutable scratch step before final model_validate

3. Phase 1 — animedex api passthrough (2 d)

See #3 for the Phase 1 runbook: live-validated endpoint catalog, drift register (kitsu.app + shikimori.one + Trace.moe quota + MangaDex Via), default UA convention (animedex/<version>), UA enforcement matrix, and the auth-fetch recipes that are deferred from this phase. Phase 1 is anonymous-only; no auth flows are wired here. The checklists below stay green-tickable in this issue, but their detail lives in #3.

3.1 Implementation order

(AniDB is excluded from Phase 1 by design; it lands in Phase 7.)

3.2 Universal flags (per plan 03 §7)

3.3 Raw passthrough method and cache semantics

The old local read-only firewall has been retired. Raw animedex api forwards caller-supplied methods and paths verbatim; animedex informs through docs and stderr warnings where needed, but does not reject raw method/path choices on the user's behalf. The remaining classifier in animedex/transport/read_only.py is advisory: it marks known-read requests as cache-eligible and makes mutating-looking or unknown raw requests bypass cache so they reach the upstream every time.

Default read shapes still matter for convenience behavior:

  • AniList: defaults to GraphQL POST / only when the user did not explicitly pass -X/--method.

  • Shikimori GraphQL: defaults to POST /api/graphql only when the user did not explicitly pass -X/--method.

  • Trace.moe: defaults to byte-body POST /search for upload search only when the user did not explicitly pass -X/--method.

  • Jikan / Kitsu / MangaDex / Danbooru / Shikimori REST / ANN / Ghibli / Quote: raw REST passthrough defaults to GET.

  • Implement and unit-test each per-backend advisory known-read/cache-eligibility rule.

  • Remove local raw method/path rejection; explicit POST, PUT, PATCH, DELETE, or any other caller-supplied method is forwarded to the upstream, with cache bypass for requests that are not classified as known reads. (Landed via PR Add raw API universal flags #17.)

3.4 Phase 1 gating

  • All eight animedex api <backend> invocations succeed against real upstreams in a one-off integration check.
  • --paginate works for the original three paginating raw backends and the later Shikimori/Quote paginated surfaces. (Landed via PR Add raw API universal flags #17.)
  • selftest() for each backend passes mock-only.

4. Phase 2 — MVP commands (2-3 d) — closes 0.1.x

4.1 Command tree

4.2 Notes

  • Every command's docstring must satisfy the policy lint.
  • Renderer is consumed verbatim from substrate; do not re-implement.
  • Trace.moe anonymous quota is 100/month; selftests must never hit the live API.

4.3 Phase 2 gating = 0.1.0 release


5. Phase 3 — Mid-tier read backends (3 d)

Notes:

  • The Danbooru tag DSL is fully documented in the docstring; rating:g is not auto-injected (plan 02).
  • MangaDex enforces UA injection and forbids Via header.
  • Waifu.im exposes --include-tag/--exclude-tag/--is-nsfw; each tag class is named in the docstring.

6. Phase 4 — Calendar / news / trivia (2 d)

  • animedex shikimori calendar/search/show/screenshots/videos/characters/staff/similar/related/external-links/topics/studios/genres/manga-search/manga-show/ranobe-search/ranobe-show/club-search/club-show/publishers/people-search/person (UA default-injected by the shared transport). (Landed via PR dev(narugo1992): add shikimori and ann high-level backends #15: high-level REST CLI/Python API with fixtures, rich models, docs, and GIF demo.)
  • animedex ann show/search/reports (XML adapter via xml.etree.ElementTree). (Landed via PR dev(narugo1992): add shikimori and ann high-level backends #15: generic lossless XML adapter, typed ANN models, warning-bearing 200 responses modeled as data, fixtures, docs, and GIF demo.)
  • animedex ghibli films/people/locations/vehicles/species (offline; bundled animedex/data/ghibli.json). (Landed via PR Add Ghibli and quote backends #14: offline snapshot-backed high-level CLI/Python API plus raw live passthrough.)
  • animedex quote random/random-by-anime/random-by-character/quotes-by-anime/quotes-by-character/anime (5 req/h, local SQLite cache, lazy-fill). (Landed via PR Add Ghibli and quote backends #14: AnimeChan v1 anonymous read surface, including paginated quotes-by-anime and quotes-by-character, with cache-before-rate-limit behaviour.)
  • selftest() for the Ghibli backend loads and validates the bundled JSON. (Landed via PR Add Ghibli and quote backends #14; Quote selftest also smokes default SQLite cache path and schema creation.)

7. Phase 5 — Aggregate commands (2 d) — closes 0.2.x

Notes:

  • The aggregate substrate is shared across all five commands: animedex/agg/_fanout.py (concurrent fan-out helper + per-source partial-failure status), animedex/models/aggregate.py (AggregateResult / AggregateSourceStatus / MergedAnime / merge_diagnostics), animedex/entry/aggregate.py (_report_failures + _report_merge_diagnostics stderr inform + _emit / _finish), animedex/utils/timezone.py. The remaining Phase 5 PRs reuse these directly rather than re-implementing.
  • Cross-source merging is the user-visible default on season (and will be on search once the follow-up merge slice lands). Matching is conservative: shared external IDs first, then a deterministic fuzzy comparison using anyascii / jaconv / unidecode for CJK/Hangul recall. The threshold (_MERGE_THRESHOLD = 70) is calibrated against the 2010-2025 adjudicated baseline at test/fixtures/aggregate/season_matrix/expected_matches.json; lowering increases recall, raising increases precision. Re-run tools/merge_eval/evaluate_rule.py after any threshold change.
  • Score merging keeps both upstream values under MergedAnime.source_details and source_payloads; do not average. Source attribution is the contract — the merged row exposes every upstream record under the sources / records map so no upstream-visible field disappears.
  • crossref consults a vendored static map first (animedex/data/crossref.json, snapshot of nattadasu/animeApi); --deep falls back to live AniDB. Until Phase 7 lands, --deep raises a typed error with a pointer to install the AniDB extra. Do not break the command surface.
  • The substrate adds runtime dependencies anyascii, jaconv, unidecode, tzdata, and python-dateutil; each is justified in PR Add calendar aggregate commands #21's commit body. The PyInstaller spec hides the lazy-load data tables for anyascii and unidecode; the animedex selftest runner now smokes both transliterators against representative CJK/kana input.
  • Tag 0.2.0 once all five commands plus the follow-up search-merge slice are green.

8. Phase 6 — MangaDex At-Home reader (1.5 d) — closes 0.3.x

  • animedex mangadex pages <chapter-id> [--save-to <dir>].

Notes:

  • Two-step flow: GET /at-home/server/{chapterId} returns a short-lived base URL; pages are fetched from <base>/data/<hash>/<file>. Do not cache the base URL across chapters.
  • HTTP/2 same-host concurrency cap of 6.
  • File naming: <page-no>.<ext> by default; user can override via flag.
  • Docstring states the DMCA reality and the agent-guidance note ("invoke only when the user explicitly asks to read manga").
  • Tag 0.3.0 once green.

9. Phase 7 — AniDB heavy track (5-8 d, independent)

Justified only because AniDB is the only source for ed2k file fingerprinting and the broadest cross-service ID map (<resources>).

Sub-tasks in dependency order:

  • Persistent rate-limit scheduler — file-backed token bucket at ~/.cache/animedex/anidb-rate-state.json. Window survives process exit. Ship this before any AniDB client code.
  • HTTP API client (gzip + XML), with retry and back-off honouring the scheduler.
  • anime-titles dump parser for offline title lookup.
  • XML→JSON adaptor that preserves source attribution.
  • UDP socket client: AUTH, ENCRYPT, sequence numbers, PING-keepalive, thread-safe in-flight map.
  • ed2k computation utility (4 KiB-block streaming hash; pure Python).
  • AniDB credential storage: keyring namespace separate from any HTTP-only client name.
  • CLI: animedex anidb show/crossref/dump-titles/fingerprint.
  • selftest() (offline) for ed2k correctness against a known-hash fixture; selftest_online() (opt-in) for HTTP liveness only.

Gating:

  • Continuous 30 min run of dump-titles + show + crossref against real AniDB without any ban.
  • Ship 0.x.x+anidb build metadata.

10. Phase 8 — Polish + Auth (3-4 d) — closes 1.0.0

OAuth UX decision (recorded 2026-05-08, see #6 for full rationale):

  • AniList OAuth Client ships with redirect_uri = https://anilist.co/api/v2/oauth/pin. Default flow is PIN (out-of-band): browser → Authorize → AniList shows code → user pastes into CLI. Zero-infrastructure on user side; cross-platform (Windows / macOS / Linux / SSH / containers).
  • client_secret ships with the binary / sdist / wheel. AniList does not support PKCE or device flow; the secret is a public client identifier with bounded leak risk (an attacker with the secret can still only issue tokens for their own account).
  • --auth-mode loopback advanced flag (http://localhost:53682/callback): smoother UX for desktop users, breaks for SSH-remote / containers; explicitly opt-in.
  • animedex auth login anilist — runs the PIN flow, stores token via animedex.auth.keyring_store.KeyringTokenStore.
  • animedex auth logout anilist — revokes via the AniList revoke endpoint, clears keyring entry.
  • animedex auth status — lists which backends have tokens stored.
  • animedex anilist viewer / notification / markdown / ani-chart-user — replace the auth-required stubs from Phase 2 with real implementations that read the token from keyring and pass it through _dispatch.call.
  • Same flow for Kitsu (password grant), Danbooru (api_key), Shikimori (OAuth code), MangaDex (Personal Client) per their per-backend recipes in Phase 1: animedex api passthrough — runbook, endpoint catalog, drift register #3 §6.

Polish (original Phase 8 scope):

  • animedex completion bash | zsh | fish (Click built-in export).
  • animedex alias set / unset / list.
  • animedex extension ... is deferred beyond v1.0; reserve the command name only.
  • --web flag for the five mappings in plan 03 §9.
  • animedex mcp serve (real MCP server entry; uses 2.9 scaffolding).
  • Sphinx docs full build, including the auto-extracted Agents Reference page.
  • pydantic JSON schema export to docs/source/schema/.

Gating = 1.0.0:

  • Three-platform make format && make test && make build && make test_cli all green.
  • python -m animedex.policy.lint green for every command.
  • Docs site live on RTD, Agents Reference page populated.
  • JSON schema documented as stable.

11. Cross-cutting concerns

11.1 Test strategy

  • test/ mirrors animedex/ exactly (one test_*.py per module).
  • Mock HTTP via responses; mock byte streams for Trace.moe.
  • make test INTEGRATION=1 opt-in for live API tests; CI does not run by default.
  • Every backend ships selftest() (offline) and may ship selftest_online() (opt-in).
  • make test_cli runs the PyInstaller artifact via subprocess.

11.2 CI matrix

  • Unit tests on Linux × Python {3.9, 3.13}; macOS × {3.11}; Windows × {3.11} (lighter coverage on the harder platforms).
  • Frozen smoke on Linux + macOS each release.
  • Windows frozen smoke deferred (PyInstaller on Windows is intermittently flaky); tracked as a separate issue.
  • make format-check + python -m animedex.policy.lint as a dedicated job.
  • make docs job that fails on Sphinx warnings.
  • Tag-triggered release job builds sdist/wheel + Linux/macOS frozen binaries and uploads to GitHub Releases + PyPI.

11.3 Documentation

  • docs/source/ is end-user-facing; it does not mention plans/ or "Top rule: human agency". The agency principle binds contributors via AGENTS.md, not library users. (Verified post-PR feat: nekos.best v2 backend + docs front-face overhaul #9: the new tutorials only carry the user-facing read of the principle — no plans/ references.)
  • docs/source/api_doc/ is regenerated via make rst_auto. (Wired in Phase 0; landing page renamed "API Reference" via PR feat: nekos.best v2 backend + docs front-face overhaul #9's auto_rst.py change.)
  • Tutorials: quickstart (4 commands) → advanced (passthrough / jq / output modes / Python library) → agent guide. (Landed via PR feat: nekos.best v2 backend + docs front-face overhaul #9: rewritten quickstart.rst with five progressive examples, tutorials/index.rst directory with per-backend deep-dives + raw_passthrough.rst + output_modes.rst + python_library.rst + agent_guide.rst. Each backend page carries its own demo GIF, References table, detailed examples, and a list-table of every endpoint with :func: cross-refs.)
  • docs/source/agents.rst extracts every --- LLM Agent Guidance --- block automatically at build time (plan 02 §4). (tutorials/agent_guide.rst documents the animedex --agent-guide flag manually; the auto-extraction directive is still TODO.)
  • docs/source/schema/ exposes the pydantic JSON schemas.

11.4 Release process

  • semver per plan 05 §7.
  • Each minor release ships a GitHub Release with sdist/wheel + Linux/macOS frozen binaries.
  • Changelog auto-derived from dev(...): summary commit messages.

12. Risk register (cross-phase)

Phase Risk Likelihood Mitigation
0 pydantic-core fails in PyInstaller medium early frozen smoke; HIDDEN_IMPORTS list
0 jq missing on Windows high graceful degradation; do not fail CI
1 AniList "degraded" 30 req/min stays permanent high conservative limiter; long cache TTL
3 MangaDex DMCA churn changes search surface medium OpenAPI regenerated each release; assert key endpoints
6 At-Home base URLs become shorter-lived medium never cache base; re-resolve per chapter
7 AniDB IP ban during development low (with scheduler) scheduler ships before client; integration tests behind opt-in flag
8 MCP protocol evolves medium use stable schema; keep API decoupled from MCP DTOs

13. First-week day-by-day (Phase 0)

Day Work
D1 AM Update setup.py / requirements.txt / classifiers / CI matrix for Python 3.9+ + pydantic v2; commit.
D1 PM Implement models/common.py + models/anime.py; selftest.
D2 AM Implement transport/http.py + transport/ratelimit.py + transport/useragent.py.
D2 PM Implement transport/read_only.py + unit tests.
D3 AM Implement cache/sqlite.py + unit tests.
D3 PM Implement auth/keyring_store.py + auth/inmemory_store.py.
D4 AM Implement render/tty.py + render/json_renderer.py + field_projection.py.
D4 PM Implement policy/lint.py + --agent-guide command.
D5 AM Implement mcp/register.py lazy stub; add Phase 0 dummy _example backend.
D5 PM Phase 0 gating: full make format && make test && make build && make test_cli; commit + tag phase-0-done.
D6-7 Phase 1: animedex api passthrough across the eight backends.

Subsequent phases will be expanded into day-by-day tracking when the previous phase's PRs are in review.


14. How to use this issue

  • The checklists above are the source of truth for "what's done vs not". Do not delete a checkbox; leave it unchecked or check it as work lands.
  • New scope additions append to the relevant phase; do not silently rewrite the gating criteria.
  • Each phase ends with a tagged release. The tag name is the phase exit criterion in §1.
  • Open questions in §0.3 must be resolved before they block the phase that needs them; otherwise, the listed default applies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions