Build the Loss of Control Observatory end to end

.env.example

@@ -0,0 +1,12 @@
+# Optional: use only after Reddit approves this project's API access
+REDDIT_CLIENT_ID=
+REDDIT_CLIENT_SECRET=
+
+# Generate with: openssl rand -hex 32
+AUTHOR_HMAC_KEY=
+
+# LLM classification credential
+OPENAI_API_KEY=
+
+# Authenticates dashboard and API access. Generate separately with: openssl rand -hex 32
+OBSERVATORY_API_KEY=

.github/workflows/ci.yml

@@ -0,0 +1,40 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+      - dev
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  quality:
+    name: Format, lint, types, and tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+      - name: Check out source
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
+        with:
+          version: "0.11.16"
+          enable-cache: true
+          cache-dependency-glob: uv.lock
+
+      - name: Install locked environment
+        run: uv sync --dev --locked
+
+      - name: Run repository checks
+        run: make check

.gitignore

@@ -0,0 +1,34 @@
+# Private project context
+background/
+
+# Secrets and local configuration
+.env
+.env.*
+!.env.example
+
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+build/
+dist/
+*.egg-info/
+
+# Local data and generated output
+data/
+artifacts/
+reports/
+*.db
+*.db-shm
+*.db-wal
+*.sqlite
+*.sqlite3
+
+# Local tools and operating systems
+.omx/
+.DS_Store

.python-version

@@ -0,0 +1 @@
+3.12

AGENTS.md

@@ -0,0 +1,131 @@
+# Engineering guide
+
+## Purpose
+
+This repository implements a compact Loss of Control Observatory based on CLTR's published pilot methodology. It collects public reports of concerning AI behaviour, classifies the evidence, stores the results with provenance, and exposes useful summaries for review.
+
+Build a small system that works end to end before adding breadth. A complete, honest vertical slice is more valuable than several unfinished components. Do not claim production readiness that has not been demonstrated.
+
+## Domain language
+
+Use these terms consistently:
+
+- **Scheming:** covert pursuit of goals that conflict with the intentions of a system's user, developer, or deployer.
+- **Scheming-related behaviour:** misalignment, covertness, or a possible precursor that may inform the study of scheming without meeting the full definition.
+- **Incident report:** a collected post that may describe scheming-related behaviour. It is an unverified report, not an incident.
+- **Credible incident report:** a report scoring at least 5 on the 0–9 evidence rubric.
+- **Incident:** a distinct real-world event represented by one or more credible reports after deduplication.
+
+Scores are prioritisation signals, not ground truth or calibrated probabilities. Public reports can support monitoring and hypothesis generation, but they cannot establish population prevalence, a model's propensity to scheme, or the mechanism behind a behaviour.
+
+## Product priorities
+
+Deliver work in this order:
+
+1. One Bluesky post flows through collection, classification, storage, and a visible report.
+2. The pipeline handles realistic batches idempotently and records provenance.
+3. Failures are isolated, observable, and recoverable.
+4. Privacy controls, retention, and erasure work and are documented.
+5. Classification quality is measured against a small human-labelled set.
+6. Add deduplication, a review dashboard, more sources, deployment, and an API only after the earlier layers are sound.
+
+When time is limited, protect the working pipeline, tests, privacy controls, recovery path, evaluation, and clear documentation. Cut interface polish and optional breadth first.
+
+## Methodology constraints
+
+- Collect public reports that combine an AI-related signal with a scheming or reaction signal. Prefer reports with direct evidence such as transcripts, screenshots, or chatbot share links.
+- Keep collection separate from classification. Preserve a seam for a cheap, high-recall pre-screen before the slower, conservative scoring stage, even if the first vertical slice uses one classifier.
+- Version every classification prompt. Store the model identifier, prompt version or content hash, timestamp, token usage, cost, score, and reasoning with each result.
+- Validate model output against a strict schema before it reaches storage. Reject out-of-range scores and malformed fields.
+- In final scoring, prefer evidence over dramatic language. Treat mundane errors, unsupported claims, jokes, promotion, deliberate jailbreaks, and user-driven misuse as common false positives. When evidence is ambiguous, choose the lower score.
+- Keep experimental or red-team results distinguishable from events reported in normal deployment. Do not mix them into real-world trend counts.
+- Count unique incidents, not posts. Deduplication must be inspectable and reversible. Guard against transitive chaining that merges distinct events across an excessive date span.
+- Choose a representative report using an explicit rule, while preserving links to every report in the group.
+- Evaluate ordinal scores with an appropriate agreement metric such as quadratic weighted Cohen's kappa. Keep the labelled sample, rubric, model outputs, and evaluation code reproducible.
+- Report collection volume and credible-incident volume separately. Normalise trend claims against relevant collection or discussion volume where possible.
+
+## Data protection
+
+Privacy controls apply before data reaches persistent storage or logs.
+
+- Never persist a raw username, display name, user ID, API token, session value, or other unnecessary identifier.
+- Replace author identifiers at collection time with `HMAC(secret_salt, stable_platform_identifier)`. Keep the salt outside the repository and database. A plain hash is not sufficient.
+- Treat `posts_raw` as immutable source evidence after mandatory redaction and minimisation. Corrections and derived data belong in separate records.
+- Store only fields needed for analysis, provenance, deduplication, retention, or erasure.
+- Apply documented retention periods to raw posts, derived data, and archived artifacts. Deletions must be auditable without retaining the deleted content.
+- Support erasure by pseudonymous author identifier and remove dependent records safely.
+- Do not send content to an LLM provider unless the configured data-handling terms are documented. Never assume an API has zero retention.
+- Do not expose collected text, artifacts, or exports publicly by default. The dashboard and export paths require explicit access controls before deployment.
+- Never commit `.env`, credentials, databases containing collected data, report artifacts containing personal data, or the ignored `background/` directory.
+
+## Architecture
+
+Keep boundaries explicit:
+
+- `collector/`: external API access, query construction, redaction, retries, and collection metadata.
+- `classifier/`: prompt loading, provider calls, response validation, scoring, and cost accounting.
+- `warehouse/`: schema, migrations, transactions, repositories, retention, erasure, and audit records.
+- `dashboard/`: read-only presentation and authenticated review actions.
+- `docs/`: decisions, operations, data protection, evaluation, and known limitations.
+- `tests/`: behaviour-focused tests and inert provider fixtures.
+
+Domain logic must not depend directly on an API SDK, web framework, or database driver. Put those dependencies behind narrow adapters so core behaviour can be tested offline.
+
+Use SQLite for the local demonstration. Keep queries and transactions disciplined enough that a later move to PostgreSQL has clear boundaries; do not build a speculative compatibility layer.
+
+## Python standards
+
+- Target one documented Python version and declare it in project metadata.
+- Use a `pyproject.toml` and a reproducible lock file. Add only dependencies that earn their maintenance cost.
+- Type public functions and domain models. Prefer small, explicit data structures over untyped dictionaries passed between layers.
+- Use UTC-aware timestamps at every boundary and store them in an unambiguous format.
+- Pass configuration into components. Do not read environment variables throughout business logic.
+- Make repeated collection and classification safe. Use stable external IDs, database uniqueness constraints, and upserts or conflict handling deliberately.
+- Use transactions for multi-record state changes. Enable SQLite foreign keys and define deletion behaviour explicitly.
+- Set timeouts on network calls. Retry only transient failures, with bounded exponential backoff and jitter. Respect rate-limit responses and provider retry hints.
+- Distinguish permanent validation failures from transient provider failures. Preserve enough structured error context for replay without leaking secrets or personal data.
+- Keep modules focused. Reuse existing code before adding helpers or abstractions. Delete dead code rather than preserving speculative paths.
+
+## Reliability and operations
+
+- Give every pipeline run a `run_id`. Emit structured JSON logs with the component, operation, outcome, duration, item counts, failure counts, and cost where relevant.
+- Never log raw credentials, author identifiers, full post bodies, full prompts containing user data, or unredacted provider responses.
+- A bad item must not terminate a batch. Record classification failures in a dead-letter queue and continue.
+- Replay must be idempotent. Record retry count, last error, and last-attempt time, and prevent concurrent workers from processing the same item.
+- Detect silent collection failures. Compare run volume with a documented baseline and make abnormal drops visible without treating them as proof of an upstream outage.
+- Provide health checks that distinguish process health from dependency readiness when a service is deployed.
+- Scheduled jobs must prevent overlapping runs or make overlap safe.
+- Document recovery steps in `RUNBOOK.md` and verify them through failure injection before presenting them as supported.
+
+## Testing and verification
+
+Tests must be deterministic and run without live API credentials.
+
+- Keep saved, redacted API responses as fixtures. Do not make network calls in unit tests.
+- Test public behaviour and database invariants, not implementation details.
+- At minimum, cover redaction before persistence, idempotent upserts, prompt-version idempotency, schema validation, score bounds, dead-letter handling and replay, cost calculation, retention, erasure, and audit logging.
+- Use integration tests for schema creation, constraints, transactions, and command-line workflows.
+- Add a regression test before fixing a defect when practical.
+- Run focused tests after each change, then the full test suite, formatting, linting, type checking, and build checks before completion.
+- Inspect the final diff for secrets, personal data, generated files, accidental API changes, and unrelated edits.
+- Do not describe a path as tested unless the relevant command was run and its result was inspected.
+
+## Git and change discipline
+
+- Keep each commit to one coherent outcome. Include its tests and necessary documentation in the same commit.
+- Do not mix formatting, refactoring, dependency changes, and behaviour changes unless they are inseparable.
+- Write the subject as the reason for the change. Add a short body when constraints or trade-offs are not obvious.
+- Record meaningful verification with a `Tested:` trailer and any known gap with `Not-tested:`.
+- Commit only a reviewed, passing state. Do not rewrite shared history or discard work that you did not create.
+- Use architecture decision records for consequential choices with credible alternatives, not for routine implementation details.
+
+## Documentation and communication
+
+Write external documentation for a non-technical reader who understands the project and has limited time. Lead with a plain-language explanation of the purpose, outcome, and limits. Add a clearly separated technical section only when it helps someone build, operate, or review the system. Internal engineering notes may assume more technical knowledge.
+
+- Use short sentences, concrete claims, and ordinary words. Remove filler, hype, generic praise, and repeated conclusions.
+- Use technical terms only when they make the statement more precise. Define project-specific terms once and use them consistently.
+- Explain why a design exists, its operational trade-offs, and how it was verified. Do not narrate obvious code.
+- Keep setup commands executable from a fresh clone.
+- Maintain a candid distinction between what was validated, what is assumed, and what would change at higher scale.
+- State methodology limitations beside results. Never present classifier output, public reporting volume, or deduplicated counts with more certainty than the evidence supports.

Makefile

@@ -0,0 +1,24 @@
+.PHONY: check format format-check install lint report test typecheck
+
+install:
+	uv sync --dev --locked
+
+format:
+	uv run ruff format .
+
+format-check:
+	uv run ruff format --check .
+
+lint:
+	uv run ruff check .
+
+typecheck:
+	uv run mypy src tests
+
+test:
+	uv run pytest
+
+report:
+	uv run observatory report
+
+check: format-check lint typecheck test