Release v1.2.0

.github/workflows/ci.yml

@@ -41,3 +41,8 @@ jobs:
         run: |
           uv run tests/test_core.py
           uv run tests/test_selection.py
+          uv run tests/test_backup.py
+          uv run tests/test_promote.py
+          uv run tests/test_layers.py
+          uv run tests/test_cli.py
+          uv run tests/test_wizard.py

.gitignore

@@ -27,3 +27,8 @@ classify_state.json
 # build artifacts
 /dist/
 *.egg-info/
+
+# tool state (desloppify scan/plan state + scorecard; octopus scratch)
+.desloppify/
+.claude-octopus/
+scorecard.png

CLAUDE.md

@@ -59,7 +59,7 @@ NOT used). All pickers return newest-added-first.
 
 **Packaging.** The code is a proper installable package under `src/scourgify/` (hatchling; on PyPI as
 `scourgify`). The single `scourgify` console command (`cli.py`) dispatches argv to the tools: bare → wizard,
-`setup`/`audit`/`apply` → wrangle, `classify`, `staleness`. Bundled `defaults/` (and `_writer.py`, `afm.swift`)
+`setup`/`audit`/`apply` → wrangle, `classify`, `staleness`, `rollback` → common. Bundled `defaults/` (and `_writer.py`, `afm.swift`)
 ship **inside** the package (read-only at runtime); per-user `config.toml`, `overrides/`, and `data/` resolve
 against the **current working directory** — so `uv run` from the repo (CWD = repo root) behaves exactly as
 before, while an installed copy writes proposals under wherever it's invoked. `common.HERE` is the package
@@ -71,9 +71,13 @@ dir (use it only for shipped read-only files); anything user-writable keys off `
 - **Writes** — the standalone tool computes the change-set, serializes it to JSON, and shells out **once** to
   `calibre-debug -e _writer.py -- ops.json` (Calibre's API is the only fast batch-write path; `calibredb set_metadata`
   is one book per process). `run_writer()` (in **common.py**; imported by wrangle/classify/staleness) does this,
-  **automatically snapshots metadata.db to `/tmp/ff_<ts>.db` first** (prints the path — that's the rollback), and
-  **refuses to run while Calibre is open** (it locks the DB). The user never types `calibre-debug`. Master rollback =
-  the full "Export all Calibre data" backup.
+  **automatically snapshots metadata.db to `data/backups/ff_<ts>.db` first** (pruned to the last 20; `scourgify
+  rollback [--list]` restores one, and the current db is snapshotted before a restore so rollback is reversible), and
+  **refuses to run while Calibre is open** (`calibre_open()` detects via pgrep→ps and fails *closed* if neither
+  exists — it locks the DB). It also **refuses, before writing, a change-set that would catastrophically empty a
+  populated column** (`_is_wipe`/`_predict_populated`: >90% of a ≥100-book column) — a coarse last-line net that
+  covers *every* writer (classify/promote/staleness/setup), not just wrangle's semantic guards; `--force` overrides.
+  The user never types `calibre-debug`. Master rollback = the full "Export all Calibre data" backup.
 - **`_writer.py`** is the only file that imports Calibre — a generic ops executor (`create_column` / `set_field` /
   `stamp_now` / `set_pref`).
 - **`common.py`** is the shared core: lazy `CALIBRE_LIBRARY` resolution (importing any module never exits),
@@ -85,8 +89,10 @@ pure core — `transform`, trope-chain resolution, `parse_resp`, the TOML reader
 `uv run tests/test_selection.py` pins the selection semantics against a throwaway sqlite `metadata.db` built
 by `tests/fixture_db.py` (covers both custom-column storage shapes). CI runs both. `scourgify audit` remains the
 against-your-library check: full new state, before/after counts, and SAFETY lines asserting **no book loses its
-last fandom or character** plus a **tag mass-deletion guardrail** (`apply` aborts if tags would shrink >25% and
->200 assignments — the signature of an over-broad junk rule; `--force` overrides).
+last fandom or character** (`apply` aborts if any book would end with an empty `#fandoms`/`#characters` it started
+with — a bad `fandoms.csv` alias→"" or an empty `decompose` payload; a blocklisted non-fandom relocated to tags is
+preserved and not counted) plus a **tag mass-deletion guardrail** (`apply` aborts if tags would shrink >25% and
+>200 assignments — the signature of an over-broad junk rule). `--force` overrides both.
 
 ## Maintenance loop (after new FanFicFare downloads)
 
@@ -130,11 +136,14 @@ concept already lives in that book's structured column (**backfill-before-strip*
 into the numbered Series field, and aggressive franchise unification (e.g. all Fate/Nasuverse → `Type-Moon`).
 
 **`classify.py` — content-based tagging** (separate from the deterministic engine; uses an LLM). Two outputs
-per book: `added_tags` (chosen from the controlled vocab — bundled `defaults/classify_vocab.txt` merged with
-the user's `overrides/classify_vocab.txt` via `load_vocab()`, lazily, so installed copies stay overridable →
-applied) and `proposed_new` (novel candidates → aggregated to `classify_newtags_ranked.csv` for
-review→promotion, so the vocab grows without freeform noise). Engines `--engine apple|claude|openai|gemini`
-(keys via env: `ANTHROPIC_/OPENAI_/GEMINI_API_KEY`); `apple` = on-device, free, single-threaded. Concurrency
+per book: `added_tags` (chosen from the controlled vocab — hand-curated `defaults/classify_vocab.txt` ∪ the
+frequency-gated AO3 seed `defaults/classify_vocab_ao3.txt` (the ~120 highest-use AO3 freeform tropes,
+generated by `build_classify_seed.py` from `data/ao3_vocab.csv` so the classifier isn't under-tagging out of
+the box — regenerable, never hand-edit it) merged with the user's `overrides/classify_vocab.txt` via
+`load_vocab()`, lazily + dedup-on-merge, so installed copies stay overridable → applied) and `proposed_new`
+(novel candidates → aggregated to `classify_newtags_ranked.csv` for review→promotion, so the vocab grows
+without freeform noise). Engines `--engine apple|claude|openai|gemini|mistral` (keys via env:
+`ANTHROPIC_/OPENAI_/GEMINI_/MISTRAL_API_KEY`; `--bakeoff` compares them on sample books); `apple` = on-device, free, single-threaded. Concurrency
 via `ThreadPoolExecutor` (`--workers`), retry/backoff, incremental save + resume. `--text-fallback` samples
 the book's own prose (EPUB via zipfile, other formats via `ebook-convert`) when the description (Calibre's
 built-in `comments` table) is too thin. Scope flags (`--incremental` / `--last N` / `--since DATE`) go through

README.md

@@ -85,6 +85,24 @@ The engine reads, first to last (later wins):
 Data from the [OTW's Selective data dump for fan statisticians](https://archiveofourown.org/admin_posts/18804)
 (2021-02-26), released for public reuse — thank you, Tag Wrangling volunteers.
 
+> ⚠️ **Taxonomy vintage — the AO3 layer is frozen at the 2021-02-26 dump.** It's the only public dump OTW
+> has released, so franchises that blew up after it (e.g. **Baldur's Gate 3**, **Elden Ring**,
+> **Honkai: Star Rail**) aren't in `defaults/ao3/` and won't auto-fold on a fresh install. The
+> override/promote loop covers the gap — the classifier still content-tags these books, and the
+> `#fandoms`/`#characters` values pass through untouched (they just aren't *unified* to a franchise
+> canonical). When OTW publishes a newer dump, `build_ao3_layer.py` regenerates the whole layer.
+>
+> **Add a post-2021 fandom yourself** — drop rows into `overrides/` (they win over everything and survive
+> upgrades; formats under [Customizing](#customizing)):
+> ```csv
+> # overrides/fandoms.csv   (alias,canonical) — unify a franchise's media splits
+> Baldur's Gate 3,Dungeons & Dragons
+> BG3,Dungeons & Dragons
+> # overrides/characters.csv (variant,canonical,fandom) — fold a character's abbreviations (blank = global)
+> Astarion Ancunín,Astarion,Dungeons & Dragons
+> ```
+> Or just let `scourgify promote` adjudicate the novel tags your classify runs surface — same effect, no hand-editing.
+
 ---
 
 ## FanFicFare → Calibre columns (how the linking works)
@@ -183,8 +201,10 @@ aborts if tags would **mass-shrink** (>25% of assignments and >200 lost — the
 over-broad `junk.txt` rule; `--force` overrides after you've checked). A redundant tag is only
 stripped when the concept already lives in that book's structured column. `audit` is read-only
 (plain `python3`, fine with Calibre open); `apply`/`setup` use the Calibre API (Calibre **closed**).
-**Every write automatically snapshots `metadata.db` to `/tmp/ff_<timestamp>.db` first** and prints
-the path — that's your instant rollback (master rollback = a full "Export all Calibre data" backup).
+**Every write automatically snapshots `metadata.db` to `data/backups/ff_<timestamp>.db` first**
+(pruned to the last 20) and prints the path — `scourgify rollback` restores the newest, or
+`scourgify rollback --list` to pick one; the current db is itself snapshotted before a restore, so a
+rollback is reversible too (master rollback = a full "Export all Calibre data" backup).
 
 ## Maintenance — after new downloads / updates
 New stories arrive **raw** (junky subject tags, unfolded names). **Order matters — deterministic
@@ -223,16 +243,21 @@ scourgify classify --apply --step                   # review each book's tags 1-
   Apple Intelligence). Ships as source; a `swift` toolchain runs it as-is, or from a checkout build the
   faster binary once: `swiftc -O src/scourgify/afm.swift -o src/scourgify/afm`. Lower quality — prone to over-tagging,
   so the prompt caps at `--max-tags 6` and dumps (>2× cap) are rejected.
-- `--engine claude|openai|gemini` — cloud APIs (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `GEMINI_API_KEY`);
-  defaults `claude-haiku-4-5` / `gpt-4o-mini` / `gemini-2.5-flash`, override with `--model`. Sharper; cheap.
+- `--engine claude|openai|gemini|mistral` — cloud APIs (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` /
+  `GEMINI_API_KEY` / `MISTRAL_API_KEY`); defaults `claude-haiku-4-5` / `gpt-4o-mini` / `gemini-2.5-flash` /
+  `mistral-small-latest`, override with `--model`. Sharper; cheap.
+- `--bakeoff` — run a few sample books through every usable engine and print the comparison, then exit
+  (writes nothing). The quick "which engine tags my library best?" check before committing to a full run.
 - **Scope — which books a run touches** (`select.py` owns this; newest-added-first, so `--batch`/`--limit`
   caps hit the new books first): `--incremental` = only new/changed books; `--last N` = the N most recently
   added; `--since DATE` = added or site-updated on/after DATE; no scope flag = books with `< --min-tags`
   (default 2) tags. Books whose description is too thin (<40 chars) are reported, not silently dropped —
   `--text-fallback` samples the book's own prose for them. Always dry-run until `--apply`.
-- The allowed tag set is the bundled `defaults/classify_vocab.txt` **plus your
-  `overrides/classify_vocab.txt`** (a line appends a term, `-term` removes one) — editable even for a
-  pipx/uv-tool install, where the bundled file lives read-only in site-packages.
+- The allowed tag set is the hand-curated `defaults/classify_vocab.txt` **∪ a frequency-gated AO3 seed**
+  (`defaults/classify_vocab_ao3.txt` — the ~120 highest-use AO3 freeform tropes, generated by
+  `build_classify_seed.py` from the OTW dump so the classifier isn't under-tagging out of the box) **plus
+  your `overrides/classify_vocab.txt`** (a line appends a term, `-term` removes one — and can trim a
+  seeded term) — editable even for a pipx/uv-tool install, where the bundled files live read-only in site-packages.
 - Long runs **save incrementally and resume** on re-run (skip books already in the proposal; `--fresh`
   to restart). `--batch N` processes only N new books per run — handy for pacing API spend/rate limits.
   Cloud engines run `--workers` requests concurrently (default 8); `apple` is single-threaded (one
@@ -286,7 +311,7 @@ un-drops, cross-column rescues) are listed for hand-editing rather than faked.
 
 ## Repo layout
 The package lives in **`src/scourgify/`**; the single `scourgify` command (`cli.py`) dispatches
-bare → wizard, `setup`/`audit`/`apply`/`overrides` → wrangle, `classify`, `staleness`, and `promote`.
+bare → wizard, `setup`/`audit`/`apply`/`overrides` → wrangle, `classify`, `staleness`, `promote`, and `rollback`.
 - **`cli.py`** — the `scourgify` entry point (argv dispatcher over the tools below)
 - **`wrangle.py`** — the engine: `setup` / `audit` / `apply`; with no command it launches the wizard
 - **`wizard.py` + `ui.py`** — the interactive wizard and its rich terminal helpers (the one

build_classify_seed.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+# Build the shipped classify vocab SEED from the AO3 freeform-tag use counts (data/ao3_vocab.csv,
+# `name,uses`, produced by build_ao3_layer.py from the OTW statistics dump).
+#
+# Why: classify's controlled vocab (defaults/classify_vocab.txt) is a tight hand-curated core, so out of
+# the box the classifier can only pick from ~120 terms and under-tags (issue #8). The highest-use AO3
+# freeforms ARE the generic, reusable tropes (Fluff, Angst, Slow Burn, Hurt/Comfort…) — a use-count
+# threshold cleanly separates them from fandom-specific noise (which never accumulates many uses). We ship
+# that frequency-gated slice as a SEPARATE file so load_vocab() = curated core ∪ AO3 seed; the curated file
+# stays hand-editable and this one stays regenerable (never hand-edit it — re-run this script).
+#
+# Maintainer-only: needs data/ao3_vocab.csv (gitignored). The OUTPUT ships in the wheel.
+#   uv run python build_classify_seed.py           # regenerate src/scourgify/defaults/classify_vocab_ao3.txt
+import csv, os, re
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+DEF = os.path.join(HERE, "src", "scourgify", "defaults")
+SRC = os.path.join(HERE, "data", "ao3_vocab.csv")
+OUT = os.path.join(DEF, "classify_vocab_ao3.txt")
+
+THRESHOLD = 20000        # AO3 use-count floor: ~125 terms, roughly doubling the curated core (tune to taste)
+
+# AO3 meta/format/medium/length tags that describe FORM, not content — noise for a content classifier.
+META = {"one shot", "drabble", "drabble collection", "podfic", "reading", "pov first person"}
+def is_meta(name: str) -> bool:
+    return name.strip().lower() in META or name.startswith("Wordcount:") or name.startswith("Podfic")
+
+def main() -> None:
+    if not os.path.exists(SRC):
+        raise SystemExit(f"{SRC} not found — run build_ao3_layer.py first (needs the OTW dump).")
+    curated = {l.strip().lower() for l in open(os.path.join(DEF, "classify_vocab.txt"))
+               if l.strip() and not l.startswith("#")}
+    rows = [(r["name"], int(r["uses"] or 0)) for r in csv.DictReader(open(SRC))]
+    seed, seen = [], set(curated)
+    for name, uses in sorted(rows, key=lambda r: -r[1]):          # highest-use first
+        if uses < THRESHOLD: break
+        if re.search(r"\([^)]*\)\s*$", name): continue           # "(Fandom)" suffix -> fandom-specific
+        if is_meta(name): continue                               # format/medium/length meta -> not content
+        if name.strip().lower() in seen: continue                # already curated (or a dup) -> skip
+        seen.add(name.strip().lower()); seed.append(name.strip())
+    with open(OUT, "w") as f:
+        f.write("# GENERATED by build_classify_seed.py — do NOT hand-edit (re-run the script; tune via overrides/).\n")
+        f.write(f"# AO3 freeform tags with >= {THRESHOLD:,} uses (OTW dump), minus fandom-specific / meta / already-curated.\n")
+        f.write("# Merged into the controlled vocab by classify.load_vocab(); trim any with a '-term' line in overrides/classify_vocab.txt.\n")
+        for t in seed: f.write(t + "\n")
+    print(f"wrote {len(seed)} seed terms (uses >= {THRESHOLD:,}) -> {os.path.relpath(OUT, HERE)}")
+
+if __name__ == "__main__":
+    main()

pyproject.toml

@@ -58,3 +58,18 @@ name = "testpypi"
 url = "https://test.pypi.org/simple/"
 publish-url = "https://test.pypi.org/legacy/"
 explicit = true
+
+# Bandit security scan (run: `bandit -c pyproject.toml -r src/scourgify`). The skipped
+# checks were each reviewed and are safe for this tool's threat model — a single-user,
+# local CLI with NO network/web input surface (the only inputs are the user's own Calibre
+# metadata.db and their own override files; there is no attacker-controlled input):
+#   B404/B603/B607 — subprocess: fixed argv lists to trusted tools (calibre-debug,
+#                    ebook-convert, swift, the local afm binary); never shell=True.
+#   B310           — urllib: hardcoded https:// LLM-API endpoint literals; the scheme is
+#                    constant, so file:/custom-scheme abuse is impossible.
+#   B608           — SQL: queries interpolate only `?,?,?` placeholders (values bound as
+#                    params) or a schema-derived integer column id — never user text.
+#   B112           — the try/except/continue in calibre_open() is a best-effort pgrep/ps
+#                    probe that intentionally falls through and fails CLOSED (see the code).
+[tool.bandit]
+skips = ["B404", "B603", "B607", "B310", "B608", "B112"]

src/scourgify/__init__.py

@@ -1,3 +1,3 @@
 """scourgify — normalize a FanFicFare-imported Calibre library."""
 
-__version__ = "1.1.0"
+__version__ = "1.2.0"