Loss of Control Observatory Mini

AI systems sometimes act against a user's instructions or hide what they have done. Reports of these events are scattered across public forums, which makes patterns difficult to see. This project is building a small observatory to collect relevant public posts, remove direct author identifiers, score the strength of the evidence, and present the results for review.

The project adapts the public-transcript method described in CLTR's Loss of Control Observatory pilot report. It is an independent reference implementation, not an official CLTR system.

What this project tests

The pilot report describes several limits. This implementation focuses on three:

1. Coverage: the pilot collected posts from X. This project adds Bluesky as a separate source through its documented public API.
2. Evidence authenticity: the pipeline records where evidence came from and will preserve supported chatbot share links with a content hash. This reduces evidence loss but cannot prove that every public report is genuine.
3. Ordinary errors: unexpected AI behaviour is not automatically scheming. A separate Stage 3 analysis compares evidence for scheming with plausible mundane failures and routes ambiguous cases for review without replacing the pilot's original score.

The wider goal is a complete data path that can be operated safely: collection, classification, storage, recovery, evaluation, and a useful review output. The project does not estimate how often AI systems scheme across all uses, and classifier scores are not ground truth.

Production-ready engineering

This is a production-quality local demonstration, not a deployed production service. The controls below are implemented and tested. A public deployment would still need managed identity, central monitoring, backups, and a tested release and rollback process.

Safe, bounded collection

• Uses Bluesky's documented public API. It does not use browser scraping, residential proxies, or an access-control bypass.
• Sets explicit limits on posts, pages, queries, request time, and retries so one run cannot grow without bound.
• Retries timeouts, rate limits, and temporary server errors with bounded exponential backoff. It records partial coverage instead of silently treating missing data as a successful run.
• Uses stable source IDs and database uniqueness constraints. Repeated collection is safe: known posts become no-ops rather than duplicate evidence.

Cost-controlled LLM pipeline

• Separates collection from classification. A lower-cost, high-recall prescreen reduces the work sent to the stronger evidence scorer.
• Requires strict structured output before a model result can enter the warehouse. Missing fields, invalid scores, and inconsistent review routes are rejected.
• Stores the exact model, prompt hash, time, reasoning, token use, and estimated cost for every result. A prompt change creates a new version instead of rewriting history.
• Preserves CLTR's original 0–9 score while separately comparing evidence for scheming with plausible ordinary failures. This keeps the published baseline available for review.

Failure isolation and recovery

• Gives every pipeline execution a unique run ID and writes the outcome to structured JSON logs and durable warehouse metrics.
• Keeps a bad post from stopping a batch. Failed items enter a dead-letter queue while successful items remain committed.
• Leases replay work so two workers cannot retry the same failure at once. Completed replays are idempotent and do not create duplicate results.
• Includes a failure-injection demonstration that forces a timeout, queues the item, recovers it, and proves that a second replay performs no work.

Data quality and privacy

• Validates required fields, parseable UTC timestamps, non-empty text, and source-specific record structure before storage.
• Replaces author identifiers with keyed HMAC values before they reach disk. It also redacts common credential shapes from collected text.
• Keeps minimised source evidence immutable. Derived results are versioned separately and linked by foreign keys with explicit deletion rules.
• Implements retention, author erasure, and append-only audit records. Screenshot storage fails closed until face redaction is available.

Observable operations

• Records source, stage, status, duration, attempted items, successful items, failures, token use, and model cost for each run.
• Exposes freshness, queue depth, success rate, latency, spend, safe application events, and warehouse integrity in a separate operations view.
• Provides distinct liveness and readiness checks. Collection-volume checks flag unusual drops without claiming that a source outage or change in incident prevalence has been proved.
• Stores operational events without request headers, query values, response bodies, prompts, post text, or direct author identifiers.

Authenticated data access

• Protects data, operations, and export endpoints with an API key. The public dashboard shell contains no collected records.
• Keeps the browser key in session storage rather than URLs, cookies, or persistent local storage. Secrets live in an ignored environment file and can be rotated by replacement and restart.
• Sends warehouse responses with Cache-Control: no-store. CSV exports omit source text, model reasoning, author identifiers, and direct source URLs, and are capped at 10,000 rows.
• Serves a versioned API and generated OpenAPI documentation. The service binds to localhost by default and documents the controls required before external exposure.

Scale and maintainability

• Tested the production warehouse path with 50,000 synthetic posts in 13.922 seconds. It rejected a second pass of 50,000 duplicates, recorded no failed items, and passed SQLite integrity checks.
• Recorded 9,105 initial inserts per second, 101.919 ms p95 batch latency, and 49.66 MiB peak memory on the documented local test machine. These are baselines, not service-level objectives.
• Uses a modular monolith with explicit collector, classifier, warehouse, API, and dashboard boundaries. External SDKs and storage drivers do not leak into the core domain logic.
• Chooses SQLite deliberately for a local, single-writer demonstration. The boundaries support a later move to PostgreSQL and queue-backed workers without a speculative compatibility layer.

Verification and change control

• Runs deterministic tests without live credentials or network calls. The suite covers privacy, idempotency, schema validation, score bounds, cost, replay, retention, erasure, and audit records.
• Runs formatting, linting, type checking, and tests in GitHub Actions on pushes and pull requests. CI dependencies are version-pinned and installed from a locked environment.
• Keeps architecture decisions, operating procedures, limitations, and live validation evidence in version control. Claims are separated into measured, assumed, and not yet demonstrated.
• Uses forward-only database migrations and small, purpose-specific commits so changes remain reviewable and reversible.

Current status

The repository now has a complete local path from privacy-first Bluesky collection through an OpenAI prescreen and detailed scorer to versioned incident grouping. An authenticated dashboard lets reviewers explore and export the data, while a separate operations view shows run health, cost, safe logs, recovery, privacy controls, and warehouse integrity. The same service exposes a versioned API with generated documentation. Each operation records a structured run lifecycle, volume, failures, duration, and model cost. Work is tracked in the project roadmap. No collected dataset is committed.

What changes with volume

At roughly 10 times this run's volume, collection should be scheduled rather than repeated immediately, with provider-aware rate limits, resumable jobs, dead-letter replay, and batched model requests. SQLite remains reasonable for one local writer, but operational metrics and cost budgets become necessary.

At roughly 100 times the volume, use a managed relational database, object storage for evidence, queue-backed workers, and explicit service health alerts. Incident deduplication must happen before review, and access control, retention, erasure, backups, and deployment rollback become release requirements rather than planned safeguards.

Technical guide

Project Structure

src/loc_observatory/
├── collector/     Source adapters, filtering, and author redaction
├── classifier/    Evidence prompts, provider calls, and result validation
├── warehouse/     SQLite schema, retention, erasure, and audit records
├── reporting/     Static reports and exports
└── dashboard/     Authenticated review interface

Detailed decisions live in docs/adr/. Data handling is described in docs/DATA_PROTECTION.md, and operating instructions are kept in RUNBOOK.md. Non-secret settings and required environment variables are documented in docs/CONFIGURATION.md. The implemented report stages and their limits are set out in docs/METHODOLOGY.md. The conditions and aggregate results from the bounded live run are in docs/VALIDATION.md. The separate 50,000-record infrastructure benchmark is documented in docs/SCALE_TEST.md. The structured logging and durable run-metrics contract is documented in docs/OBSERVABILITY.md. Dashboard and API use, including their security limits, is documented in docs/DASHBOARD_AND_API.md.

Bluesky access is documented in docs/BLUESKY_ACCESS.md. The reason Reddit is not the live source is recorded in docs/adr/0002-use-bluesky-for-live-collection.md.

Set up the project

Requirements:

• Python 3.12
• uv
• GNU Make, or run the corresponding uv commands from Makefile

Install the development environment:

uv sync --dev --locked

Run every local quality check:

make check

Create or upgrade the ignored local warehouse:

uv run observatory warehouse migrate

After setting AUTHOR_HMAC_KEY in the ignored .env file, collect one bounded Bluesky batch:

uv run observatory bluesky collect

After setting OPENAI_API_KEY, run the pilot-style high-recall screen and then score the high results with the versioned detailed rubric:

uv run observatory prescreen
uv run observatory classify
uv run observatory classify-hypotheses --limit 5
uv run observatory incidents analyze

Set a separate OBSERVATORY_API_KEY of at least 32 characters, then start the local dashboard and API:

uv run observatory dashboard

Open http://127.0.0.1:8000 for the dashboard or http://127.0.0.1:8000/docs for the interactive API contract. Data endpoints require the key in the X-API-Key header. The service is local by default; review the deployment limits before binding it to an external interface.

Generate the self-contained report:

make report

See Evidence classification for what the score means and what it cannot establish.

The individual commands are:

make format-check
make lint
make typecheck
make test

Tests must run without live API credentials or network access.