diff --git a/.agents/skills/release/SKILL.md b/.agents/skills/release/SKILL.md new file mode 100644 index 0000000..56b9b7f --- /dev/null +++ b/.agents/skills/release/SKILL.md @@ -0,0 +1,121 @@ +--- +name: release +description: Commit, push, tag, and verify a Letterboxarr release. Use when asked to release or publish a patch, minor, or major version of this repository. +--- + +Take the working tree from where it is to a published release: one commit, a +push to `main`, an annotated tag, and the Docker images and GitHub Release the +tag builds. + +The user's request says which part of the version moves. It must be `patch`, +`minor` or `major`. **If the user says none of those, ask which — do not +guess.** The keyword decides the number; it does not decide whether the +release is worth cutting. + +## 1. Preflight + +Stop and say why if any of these does not hold: + +```bash +git branch --show-current # must be main +git fetch --tags origin +git rev-list --left-right --count origin/main...main # must not be behind +git status --short +git tag --sort=-v:refname | head -1 +``` + +- **On `main`.** Tags are cut from it, so a release from anywhere else tags a + commit the published image will not match. +- **Not behind `origin/main`.** Rebase or merge first; a tag on a stale main + ships without whatever landed in between. +- **Something to release.** Either changes to commit, or a clean tree whose + HEAD is not yet tagged. A clean tree at an already-tagged commit means there + is nothing to do — say so rather than cutting an empty release. +- **Nothing secret staged.** `.env` is gitignored and must stay that way. + +## 2. Commit + +The repo's convention, visible in `git log`: + +- **Subject**: `feat:`, `fix:` or `chore:`, then a lowercase phrase saying what + changed from the point of view of someone using it. No trailing period. Not + "update X" — what the change does, e.g. *fix: leave a film off the upcoming + tab once it is out where you are*. +- **Body**: prose paragraphs, not bullets. What was wrong and what it looked + like from the outside, then what it does now and why that is the right rule, + then whatever else moved with it. Name real examples — an actual film, an + actual list, an actual number. Wrap at 76 columns. +- **Trailer**: Preserve any trailers the user supplied. Do not add an + agent-specific co-author trailer unless the user asks for one. + +The prefix and the keyword usually agree — `feat:` with `minor`, `fix:` with +`patch`. **If they disagree, say so and ask** before committing. A `feat:` +released as a patch is usually one of them being wrong. + +Write the message to a file or heredoc rather than a `-m` string, so the body +keeps its paragraphs. + +## 3. Version + +From the latest `vX.Y.Z` tag: + +| keyword | from `v1.7.4` | +|---------|---------------| +| `patch` | `v1.7.5` | +| `minor` | `v1.8.0` | +| `major` | `v2.0.0` | + +## 4. Push and tag + +```bash +git push origin main +git tag -a vX.Y.Z -m "" +git push origin vX.Y.Z +``` + +Tags are **annotated**, never lightweight. The message is the commit subject +with the `feat:`/`fix:`/`chore:` prefix dropped and the first letter +capitalised — *Leave a film off the upcoming tab once it is out where you are*. + +Push `main` before the tag so the tag build never runs against a commit the +branch has not got. + +## 5. Watch the publish + +Both pushes trigger `.github/workflows/publish.yml`, and they publish different +things — both need to pass: + +- **the tag push** → images `X.Y.Z`, `X.Y`, `X`, plus the GitHub Release +- **the `main` push** → the `latest` image + +```bash +gh run list --limit 2 --json databaseId,headBranch,status +gh run watch --exit-status +``` + +Report both conclusions. On a failure, name the failing step: + +```bash +gh run view --json jobs -q '.jobs[].steps[] | select(.conclusion=="failure") | "\(.number). \(.name)"' +gh run view --log-failed +``` + +Do not call the release done until both runs are green. The git tag existing +proves nothing about the image — login is step 5 of 8, so a credential failure +leaves a tag with nothing published behind it. + +## Known failures + +**`unauthorized: personal access token is expired`** at *Log in to Docker Hub*. +The `DOCKERHUB_TOKEN` repository secret has lapsed. It needs a new Docker Hub +PAT with **Read & Write** scope, from +, set with +`gh secret set DOCKERHUB_TOKEN`. Then `gh run rerun ` on both runs — they +read the secret at run time, so nothing needs re-pushing or re-tagging. + +**`Resource not accessible by personal access token`** from `gh run rerun`. +The `GITHUB_TOKEN` in the environment is underscoped or stale — re-running +needs **Actions: read and write**. The shell inherits its value from whatever +launched the session, so a token rotated since then is not visible: re-source +the shell's secrets file and retry before concluding the token itself is wrong. +Failing that, re-run the jobs from the run's page on GitHub. diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 0dea7e0..e569641 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -3,118 +3,5 @@ description: Commit, push and tag a release — patch, minor or major argument-hint: patch | minor | major --- -Take the working tree from where it is to a published release: one commit, a -push to `main`, an annotated tag, and the Docker images and GitHub Release the -tag builds. - -`$ARGUMENTS` says which part of the version moves. It is one of `patch`, -`minor` or `major`. **If it says none of those, ask which — do not guess.** The -keyword decides the number; it does not decide whether the release is worth -cutting. - -## 1. Preflight - -Stop and say why if any of these does not hold: - -```bash -git branch --show-current # must be main -git fetch --tags origin -git rev-list --left-right --count origin/main...main # must not be behind -git status --short -git tag --sort=-v:refname | head -1 -``` - -- **On `main`.** Tags are cut from it, so a release from anywhere else tags a - commit the published image will not match. -- **Not behind `origin/main`.** Rebase or merge first; a tag on a stale main - ships without whatever landed in between. -- **Something to release.** Either changes to commit, or a clean tree whose - HEAD is not yet tagged. A clean tree at an already-tagged commit means there - is nothing to do — say so rather than cutting an empty release. -- **Nothing secret staged.** `.env` is gitignored and must stay that way. - -## 2. Commit - -The repo's convention, visible in `git log`: - -- **Subject**: `feat:`, `fix:` or `chore:`, then a lowercase phrase saying what - changed from the point of view of someone using it. No trailing period. Not - "update X" — what the change does, e.g. *fix: leave a film off the upcoming - tab once it is out where you are*. -- **Body**: prose paragraphs, not bullets. What was wrong and what it looked - like from the outside, then what it does now and why that is the right rule, - then whatever else moved with it. Name real examples — an actual film, an - actual list, an actual number. Wrap at 76 columns. -- **Trailer**: `Co-Authored-By: Claude Opus 5 (1M context) ` - -The prefix and the keyword usually agree — `feat:` with `minor`, `fix:` with -`patch`. **If they disagree, say so and ask** before committing. A `feat:` -released as a patch is usually one of them being wrong. - -Write the message to a file or heredoc rather than a `-m` string, so the body -keeps its paragraphs. - -## 3. Version - -From the latest `vX.Y.Z` tag: - -| keyword | from `v1.7.4` | -|---------|---------------| -| `patch` | `v1.7.5` | -| `minor` | `v1.8.0` | -| `major` | `v2.0.0` | - -## 4. Push and tag - -```bash -git push origin main -git tag -a vX.Y.Z -m "" -git push origin vX.Y.Z -``` - -Tags are **annotated**, never lightweight. The message is the commit subject -with the `feat:`/`fix:`/`chore:` prefix dropped and the first letter -capitalised — *Leave a film off the upcoming tab once it is out where you are*. - -Push `main` before the tag so the tag build never runs against a commit the -branch has not got. - -## 5. Watch the publish - -Both pushes trigger `.github/workflows/publish.yml`, and they publish different -things — both need to pass: - -- **the tag push** → images `X.Y.Z`, `X.Y`, `X`, plus the GitHub Release -- **the `main` push** → the `latest` image - -```bash -gh run list --limit 2 --json databaseId,headBranch,status -gh run watch --exit-status -``` - -Report both conclusions. On a failure, name the failing step: - -```bash -gh run view --json jobs -q '.jobs[].steps[] | select(.conclusion=="failure") | "\(.number). \(.name)"' -gh run view --log-failed -``` - -Do not call the release done until both runs are green. The git tag existing -proves nothing about the image — login is step 5 of 8, so a credential failure -leaves a tag with nothing published behind it. - -## Known failures - -**`unauthorized: personal access token is expired`** at *Log in to Docker Hub*. -The `DOCKERHUB_TOKEN` repository secret has lapsed. It needs a new Docker Hub -PAT with **Read & Write** scope, from -, set with -`gh secret set DOCKERHUB_TOKEN`. Then `gh run rerun ` on both runs — they -read the secret at run time, so nothing needs re-pushing or re-tagging. - -**`Resource not accessible by personal access token`** from `gh run rerun`. -The `GITHUB_TOKEN` in the environment is underscoped or stale — re-running -needs **Actions: read and write**. The shell inherits its value from whatever -launched the session, so a token rotated since then is not visible: re-source -the shell's secrets file and retry before concluding the token itself is wrong. -Failing that, re-run the jobs from the run's page on GitHub. +Read and follow `.agents/skills/release/SKILL.md`. The requested version bump +is `$ARGUMENTS`. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..4c2b50c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,168 @@ +# Agent instructions + +This file provides guidance to coding agents working in this repository. + +## What this is + +Letterboxarr scrapes Letterboxd lists and feeds the films on them to Radarr. It +is one FastAPI process (`main.py`, port 7373) that serves both the JSON API and +the built React SPA, with a background thread doing the crawling and syncing. +Deployed as a single Docker image. + +## Commands + +```bash +# Run the server (serves API + frontend/build at http://localhost:7373) +python main.py + +# Frontend: build, then let the backend serve it +cd frontend && npm install && npm run build + +# Frontend typecheck — the only static check in the repo +cd frontend && npx tsc --noEmit + +# Docker +docker build -t letterboxarr . +``` + +**There is no test suite** — no pytest, no `*.test.tsx`, and `react-scripts +test` has nothing to run. Verify changes by exercising the real thing: run the +server, or import the module and call the function against real scraped data. +When you touch a pure function, a throwaway script comparing its output across +cases is the expected level of rigour. + +`cd frontend && npm start` will serve the UI on :3000 but **its API calls will +404** — `package.json` has no `proxy` field and axios uses a relative +`baseURL: '/api'`. Build and let the backend serve it. + +## Importing the backend has side effects + +`lib_api.py` constructs its `LetterboxarrAPIContext` singleton **at module +import time** (`context = LetterboxarrAPIContext()` at the bottom of the class +definitions). Importing `lib_api` for any reason — including to unit-test one +pure function — loads `config.yml`, opens/migrates `./data/letterboxarr.db`, +and starts the background sync thread against the live Letterboxd and Radarr. + +For a quick check of a pure helper, this is usually tolerable (the thread dies +with the process), but know that it happens and never do it against data you +care about. + +## Architecture + +Data flows in one direction, and every read the UI does stops at SQLite: + +``` +config.yml ──> lib_config watch items (Letterboxd paths + per-list filters/tags) + │ + v + lib_letterboxd scraper: curl_cffi impersonation + BeautifulSoup + │ + v + lib_db SQLite — the source of truth, not a cache + │ + ┌──────────┴──────────┐ + v v + lib_radarr lib_api FastAPI routes, JWT auth, serves the SPA + (adds movies) (reads stored data only) +``` + +`lib_sync.LetterboxarrSync.sync_once()` is the round, driven on the configured +interval by `LetterboxarrThread`: refresh the listings, hand new films to +Radarr, then read release tables, then read ratings. `lib_refresh.ListRefresher` +owns all the "keep the stored data fresh" logic. + +### The database is the application's data, not a cache + +This is the single most important idea in the codebase and it is why +`lib_db.py` has no expiry anywhere. API reads answer from SQLite; the +background refresher replaces a stored listing **only once its replacement has +been read in full**. A crawl that is slow, refused or rate-limited therefore +degrades into serving yesterday's list rather than serving nothing or, worse, +serving a half-read list as if films had left it. + +Consequences worth internalising before changing scraper or refresher code: + +- A partial crawl must raise, not return what it got. Returning a short list + silently overwrites a complete one, which reads downstream as films having + been removed — and auto-add reacts to that. +- Endpoints never crawl. Opening a page must not wait on Letterboxd. If you + need data the UI doesn't have, the fix goes in `ListRefresher`, not the route. + +### Crawl budgets + +Letterboxd rate-limits and bot-blocks, so every request goes through a single +`crawl_lock` — no two crawls ever run concurrently — and the paging loops sleep +a second between pages. Listings are a page per hundred films; +release tables and ratings are a page *per film*, so they are budgeted +separately in `lib_refresh.py`: + +| | max age | reads per round | +|---|---|---| +| Release tables | 12 h | 100 | +| Ratings | 30 d | 500 | + +Anything left over is logged and picked up by later rounds. Raising these has a +direct wall-clock cost on every sync round — the constants carry the reasoning +in their comments. + +### Scraper specifics (`lib_letterboxd.py`) + +- **`curl_cffi`, not `requests`**, for browser TLS impersonation. Fingerprints + are tried in order because Letterboxd refuses some of them on member pages + (a 403 on page 2 while page 1 answers fine). +- **Categories overlap** (`film`, `short_film`, `documentary`, `tv_show`, + `unreleased`) so `CATEGORY_SKIP_FILTERS` is ordered and first match wins, + with `unreleased` first. +- **Dates are parsed against a `MONTHS` table, not `strptime`** — `%b` follows + the process locale, and a base image that set one would silently stop reading + every date on the page. +- **Watch items accept a path or a whole URL.** A privately shared list is only + reachable through its secret `boxd.it` link; its ordinary + `//list//` URL 404s for everyone but the owner. +- Posters must be read from the main column, not the whole document — a cloned + list shows its source's posters in the sidebar on every page. + +## Configuration + +`config.yml` (gitignored; see `examples/config.example.yml`) is the only live +configuration path — edited through the UI as well as by hand. Two traps: + +- **`.env` in the repo root is not read by the application.** Nothing imports + `python-dotenv`, and `lib_config.load_config_from_env()` — which reads + `RADARR_*`, `LETTERBOXD_USERNAME`, `SYNC_INTERVAL_MINUTES` — **has no call + sites and is dead legacy code**. Those variables are for docker-compose and + shell use only. Changing them changes nothing about a running app. +- The env vars that *are* live are read by `lib_api.py` at import: + `SECRET_KEY`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`. All three have insecure + defaults. + +`letterboxd.country` matters more than it looks: it is spelled the way +Letterboxd spells it in a film's releases table (`USA`, `UK`, `France`, +`Czechia`), and it drives the whole Upcoming tab. + +## Code style + +The prose in this codebase is load-bearing and quite specific — match it rather +than defaulting to house style. + +- **Comments and docstrings say *why*, in full sentences**, and name the + concrete failure they prevent ("a 403 on page two of a 264-film list cut it + to a hundred"). They do not restate what the code does. A rule with a + non-obvious edge gets a paragraph explaining the edge, not a bullet list. +- **Docstring first line is a phrase, not a sentence** — "The release a film is + dated by, None when it has none still to come". +- Real examples over abstractions: an actual film, an actual count. +- No emoji in code or comments. Frontend copy is sentence case and explains + itself to the user (see the empty states in `UpcomingPage.tsx`, which + distinguish four reasons a page can be empty). + +## Releases + +Use the `release` skill with a `patch`, `minor` or `major` bump. It runs commit +→ push → annotated tag and watches the builds. See +`.agents/skills/release/SKILL.md` for the conventions and known credential +failures. + +Pushing a `v*.*.*` tag publishes to Docker Hub and cuts a GitHub Release; a +push to `main` moves `latest`. **Both** runs must pass — the Docker Hub login +is step 5 of 8, so a lapsed credential leaves a tag with no image behind it. diff --git a/CLAUDE.md b/CLAUDE.md index 51c29b3..43c994c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,167 +1 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## What this is - -Letterboxarr scrapes Letterboxd lists and feeds the films on them to Radarr. It -is one FastAPI process (`main.py`, port 7373) that serves both the JSON API and -the built React SPA, with a background thread doing the crawling and syncing. -Deployed as a single Docker image. - -## Commands - -```bash -# Run the server (serves API + frontend/build at http://localhost:7373) -python main.py - -# Frontend: build, then let the backend serve it -cd frontend && npm install && npm run build - -# Frontend typecheck — the only static check in the repo -cd frontend && npx tsc --noEmit - -# Docker -docker build -t letterboxarr . -``` - -**There is no test suite** — no pytest, no `*.test.tsx`, and `react-scripts -test` has nothing to run. Verify changes by exercising the real thing: run the -server, or import the module and call the function against real scraped data. -When you touch a pure function, a throwaway script comparing its output across -cases is the expected level of rigour. - -`cd frontend && npm start` will serve the UI on :3000 but **its API calls will -404** — `package.json` has no `proxy` field and axios uses a relative -`baseURL: '/api'`. Build and let the backend serve it. - -## Importing the backend has side effects - -`lib_api.py` constructs its `LetterboxarrAPIContext` singleton **at module -import time** (`context = LetterboxarrAPIContext()` at the bottom of the class -definitions). Importing `lib_api` for any reason — including to unit-test one -pure function — loads `config.yml`, opens/migrates `./data/letterboxarr.db`, -and starts the background sync thread against the live Letterboxd and Radarr. - -For a quick check of a pure helper, this is usually tolerable (the thread dies -with the process), but know that it happens and never do it against data you -care about. - -## Architecture - -Data flows in one direction, and every read the UI does stops at SQLite: - -``` -config.yml ──> lib_config watch items (Letterboxd paths + per-list filters/tags) - │ - v - lib_letterboxd scraper: curl_cffi impersonation + BeautifulSoup - │ - v - lib_db SQLite — the source of truth, not a cache - │ - ┌──────────┴──────────┐ - v v - lib_radarr lib_api FastAPI routes, JWT auth, serves the SPA - (adds movies) (reads stored data only) -``` - -`lib_sync.LetterboxarrSync.sync_once()` is the round, driven on the configured -interval by `LetterboxarrThread`: refresh the listings, hand new films to -Radarr, then read release tables, then read ratings. `lib_refresh.ListRefresher` -owns all the "keep the stored data fresh" logic. - -### The database is the application's data, not a cache - -This is the single most important idea in the codebase and it is why -`lib_db.py` has no expiry anywhere. API reads answer from SQLite; the -background refresher replaces a stored listing **only once its replacement has -been read in full**. A crawl that is slow, refused or rate-limited therefore -degrades into serving yesterday's list rather than serving nothing or, worse, -serving a half-read list as if films had left it. - -Consequences worth internalising before changing scraper or refresher code: - -- A partial crawl must raise, not return what it got. Returning a short list - silently overwrites a complete one, which reads downstream as films having - been removed — and auto-add reacts to that. -- Endpoints never crawl. Opening a page must not wait on Letterboxd. If you - need data the UI doesn't have, the fix goes in `ListRefresher`, not the route. - -### Crawl budgets - -Letterboxd rate-limits and bot-blocks, so every request goes through a single -`crawl_lock` — no two crawls ever run concurrently — and the paging loops sleep -a second between pages. Listings are a page per hundred films; -release tables and ratings are a page *per film*, so they are budgeted -separately in `lib_refresh.py`: - -| | max age | reads per round | -|---|---|---| -| Release tables | 12 h | 100 | -| Ratings | 30 d | 500 | - -Anything left over is logged and picked up by later rounds. Raising these has a -direct wall-clock cost on every sync round — the constants carry the reasoning -in their comments. - -### Scraper specifics (`lib_letterboxd.py`) - -- **`curl_cffi`, not `requests`**, for browser TLS impersonation. Fingerprints - are tried in order because Letterboxd refuses some of them on member pages - (a 403 on page 2 while page 1 answers fine). -- **Categories overlap** (`film`, `short_film`, `documentary`, `tv_show`, - `unreleased`) so `CATEGORY_SKIP_FILTERS` is ordered and first match wins, - with `unreleased` first. -- **Dates are parsed against a `MONTHS` table, not `strptime`** — `%b` follows - the process locale, and a base image that set one would silently stop reading - every date on the page. -- **Watch items accept a path or a whole URL.** A privately shared list is only - reachable through its secret `boxd.it` link; its ordinary - `//list//` URL 404s for everyone but the owner. -- Posters must be read from the main column, not the whole document — a cloned - list shows its source's posters in the sidebar on every page. - -## Configuration - -`config.yml` (gitignored; see `examples/config.example.yml`) is the only live -configuration path — edited through the UI as well as by hand. Two traps: - -- **`.env` in the repo root is not read by the application.** Nothing imports - `python-dotenv`, and `lib_config.load_config_from_env()` — which reads - `RADARR_*`, `LETTERBOXD_USERNAME`, `SYNC_INTERVAL_MINUTES` — **has no call - sites and is dead legacy code**. Those variables are for docker-compose and - shell use only. Changing them changes nothing about a running app. -- The env vars that *are* live are read by `lib_api.py` at import: - `SECRET_KEY`, `ADMIN_USERNAME`, `ADMIN_PASSWORD`. All three have insecure - defaults. - -`letterboxd.country` matters more than it looks: it is spelled the way -Letterboxd spells it in a film's releases table (`USA`, `UK`, `France`, -`Czechia`), and it drives the whole Upcoming tab. - -## Code style - -The prose in this codebase is load-bearing and quite specific — match it rather -than defaulting to house style. - -- **Comments and docstrings say *why*, in full sentences**, and name the - concrete failure they prevent ("a 403 on page two of a 264-film list cut it - to a hundred"). They do not restate what the code does. A rule with a - non-obvious edge gets a paragraph explaining the edge, not a bullet list. -- **Docstring first line is a phrase, not a sentence** — "The release a film is - dated by, None when it has none still to come". -- Real examples over abstractions: an actual film, an actual count. -- No emoji in code or comments. Frontend copy is sentence case and explains - itself to the user (see the empty states in `UpcomingPage.tsx`, which - distinguish four reasons a page can be empty). - -## Releases - -Use `/release patch|minor|major` — it runs commit → push → annotated tag and -watches the builds. See `.claude/commands/release.md` for the conventions and -the known credential failures. - -Pushing a `v*.*.*` tag publishes to Docker Hub and cuts a GitHub Release; a -push to `main` moves `latest`. **Both** runs must pass — the Docker Hub login -is step 5 of 8, so a lapsed credential leaves a tag with no image behind it. +@AGENTS.md