animetaScraper

Scrapes AniList's public GraphQL API for anime metadata and publishes a versioned catalog (manifest + SQLite checkpoints + JSON deltas) as static files. This is the single source of truth for the shared anime catalog that anigit's refresh command downloads and applies to each user's local SQLite cache.

This project does not track anything personal — no watch history, no per-user data. Just: AniList's public catalog → normalized SQLite rows → published as static files, on a schedule, from one VM.

For the full design rationale behind every decision below (why Python, why no self-hosting, why this exact schema, etc.), see brainstorm.md. This README is the "how to run it" reference; brainstorm.md is the "why it's built this way" reference.

Requirements

• Python 3.12+
• uv for dependency management

Setup

uv sync

This creates a .venv/ and installs dependencies (requests) from the lockfile.

Running a scrape

All commands are run via uv run, which uses the project's managed environment automatically.

Full catalog scrape

uv run python src/scrape.py --served-dir /path/to/output

Scrapes every year from 1960 through next year, year by year (working around AniList's 5,000-item-per-query cap — see brainstorm.md 1.3). Takes roughly 10–35 minutes depending on AniList's current rate limit (see brainstorm.md 1.9). Publishes either a full checkpoint or a delta into --served-dir, depending on the current run number (see "How runs work" below).

Test mode (single year)

uv run python src/scrape.py --served-dir /path/to/output --test --test-year 2026

Scrapes just one year. Useful for quick local iteration or validating that AniList's schema still matches what this project expects, without waiting through a full multi-year scrape.

How runs work

Each invocation of scrape.py is one "run." Runs are numbered sequentially, and the number is derived entirely from manifest.json in the served directory — there's no separate counter file. The very first run against an empty/nonexistent --served-dir is run 1.

• Every 5th run (5, 10, 15, ...) publishes a full SQLite checkpoint — a complete snapshot of the catalog at that point in time.
• Every other run publishes a delta — just the rows that changed since the last run (AniList exposes no "updated since" filter, so this project always re-scrapes everything and diffs locally; see brainstorm.md 1.7).
• There's no separate "previous state" file kept anywhere. Each run reconstructs "what we currently know" by reading its own most recent published checkpoint and replaying every delta since — the published output is the source of truth (see brainstorm.md 1.14). This is deliberately the same logic anigit refresh needs client-side.

Because of this, running the scraper twice against the same --served-dir produces run 2, not a second run 1 — the manifest already recorded the first run. Use a fresh, empty directory if you want a clean "first ever run."

Running on the VM

The scraper runs on the project's Oracle Cloud VM (see brainstorm.md 1.10 for VM specs). This is the process for a manual run there; automated scheduling via cron is not set up yet (see "Not yet built").

One-time setup

Clone the repo and sync dependencies, same as local setup:

git clone git@github.com:TSSS-org/animetaScraper.git
cd animetaScraper
uv sync

Create the served directory, owned by your own user rather than root (the scraper must run as a normal user, never root — see brainstorm.md 1.24):

sudo mkdir -p /srv/animeta
sudo chown $USER:$USER /srv/animeta

This only needs to happen once. /srv/animeta is the real, live served directory (see brainstorm.md 1.20) — this is where production output actually lives, separate from any local fixtures/ test runs.

Running a scrape detached

A full scrape takes 10–35 minutes (see brainstorm.md 1.9), so it should survive an SSH disconnect rather than dying with the session. Using nohup:

nohup uv run python src/scrape.py --served-dir /srv/animeta > scrape_run.log 2>&1 &

This detaches the process, and redirects all output (logs and errors alike) into scrape_run.log instead of your terminal. You'll get back a line like [1] 2815342 confirming it started — that's the background job number and process ID, not an error.

Check progress:

tail -f scrape_run.log

(Ctrl+C only stops watching the log — it does not stop the actual scrape, since nohup already detached it from this terminal.)

Check it's still running:

ps aux | grep scrape.py

You can safely close the SSH session entirely once the process has started; it keeps running on the VM regardless, and you can reconnect later to check scrape_run.log or run the ps aux check again.

Project layout

src/
  anilist_client.py   # Talks to AniList: query shape, pagination, rate-limit
                       # pacing, retry/backoff policy
  diff.py              # Converts raw AniList data into our row shape, cleans
                       # up description text, reconstructs previous state,
                       # computes what changed
  publish.py           # Atomic file writes, manifest management, checkpoint
                       # SQLite generation
  scrape.py            # Entrypoint — wires the above together into one run

fixtures/
  README.md            # How to (re)generate local dev/test fixtures
                       # (gitignored — not committed, can grow to tens of MB)

brainstorm.md           # Full design doc: every decision and the reasoning
                       # behind it, in the order they were made

Output format

Published output looks like this inside --served-dir:

manifest.json
snapshots/
  checkpoint-005.sqlite
  checkpoint-010.sqlite
deltas/
  delta-001.json
  delta-002.json
  ...

manifest.json lists every checkpoint and delta published so far, plus the latest run number. See brainstorm.md section 2 for the exact JSON shapes and the SQLite column schema.

Ordering guarantee: the delta/checkpoint file for a run is always fully written before manifest.json is updated to reference it. A crash mid-run can never leave manifest.json pointing at a file that doesn't exist — the run simply doesn't appear as having happened, and the next scheduled run covers the gap.

Known data quirks

• AniList's description(asHtml: false) does not strip manually-embedded <br> tags that show up in some entries' source text. This project converts them to paragraph breaks and strips any other stray HTML, normalizing surrounding whitespace afterward. See brainstorm.md 1.25 if you're debugging description text and it looks different from what AniList's own site shows.
• AniList's API is occasionally taken fully offline for stability maintenance (returns HTTP 403 with an explicit message when this happens). This is expected and outside this project's control — a run during an outage will fail every request, log it, and publish an empty delta rather than crash. Just try again later.

Not yet built

• Automated scheduling — cron entry, run.sh wrapper (see brainstorm.md 1.20) so runs happen every 6 days without a manual SSH session. Manual runs on the VM work today (see "Running on the VM" above); this is just the automation layer on top.
• Caddy — not installed/configured yet, so /srv/animeta isn't actually served over HTTPS anywhere yet, even though the scraper can already publish correctly-shaped output into it.
• DNS — api.anigit.com doesn't resolve to anything yet.
• A generator script for synthetic "second run" dev fixtures — currently a manual process (see fixtures/README.md).