Black Lake Slack Ledger

A one-shot, cron-safe local CLI that reads an explicit allowlist of Slack channels and maintains a private append-only transcript ledger. The repository is designed to be cloned, configured, and invoked by any harness that can schedule one-shot commands.

The tool is intentionally narrow:

• Slack is read-only; the HTTP client has a method allowlist and cannot call Slack write APIs.
• The workspace and channel ID/name pairs form an explicit configuration allowlist and are verified against Slack before collection.
• Slack content is untrusted data. It is never interpreted or emitted in run summaries.
• Canonical monthly JSONL is append-only and hash-chained per channel.
• state.json and index.sqlite3 are derived and may be rebuilt without rewriting JSONL.
• Every run fails closed unless history, thread discovery, and thread pages are exhausted.
• stdout contains one JSON object; the process uses nonzero exits for failures and never prompts.

Requirements

• Python 3.11 or newer.
• A Slack user OAuth token supplied through the environment variable named in the config example (SLACK_LEDGER_USER_TOKEN by default).
• Read scopes: channels:read, channels:history, and search:read.
• The token's user must be able to access every configured channel.
• A private local directory for the ledger.

A user token is required because complete polling-mode discovery of new replies on old thread roots uses search.messages; Slack documents user tokens as acting with the user's visibility and identifies search.messages as a legacy API. The tool never writes through that token. See Slack's token documentation, search.messages, conversations.history, and conversations.replies.

Slack applies substantially lower history/replies limits to some commercially distributed non-Marketplace apps, while internal customer-built apps retain higher limits. The client honors Retry-After and uses bounded retries; review Slack's rate-limit change notice before choosing an app model.

Install or run from Git

The repository has no runtime package dependencies. A harness may run it directly from a clone:

git clone <repository-url> black-lake-slack-ledger
cd black-lake-slack-ledger
cp config.example.toml config.toml
python -m black_lake_slack_ledger doctor --offline \
  --config ./config.toml \
  --ledger-root /var/lib/black-lake-slack-ledger

Or install the console entrypoint from a local clone or Git URL:

python -m pip install --no-deps .
black-lake-slack-ledger --version

For a single-file artifact with no packaging backend, build a standard-library zipapp:

python scripts/build_zipapp.py --output dist/black-lake-slack-ledger.pyz
python dist/black-lake-slack-ledger.pyz doctor --offline --config ./config.toml --ledger-root /var/lib/black-lake-slack-ledger

Do not put a real token in TOML, command arguments, the Git repository, or scheduler logs. Inject it through the environment using the scheduler's secret facility.

Commands

black-lake-slack-ledger run --config PATH --ledger-root PATH [--mode auto|backfill|full|incremental]
black-lake-slack-ledger verify --config PATH --ledger-root PATH
black-lake-slack-ledger rebuild-index --config PATH --ledger-root PATH
black-lake-slack-ledger doctor --config PATH --ledger-root PATH [--offline]

run --mode auto follows the configured policy:

1. Missing state or an incomplete channel selects backfill.
2. The configured weekly day and time (Sunday at 00:30 UTC in the example) selects full.
3. Other runs select incremental, subtracting 259,200 seconds from each channel checkpoint.

Use a forced mode only for an operator-approved recovery or validation run.

Cron

Set the token in the cron daemon's secret-aware environment, then use absolute paths:

CRON_TZ=UTC
30 0,12 * * * cd /opt/black-lake-slack-ledger && /usr/bin/python3 -m black_lake_slack_ledger run --config /etc/black-lake-slack-ledger/config.toml --ledger-root /var/lib/black-lake-slack-ledger

The same one-shot command works in systemd timers, Kubernetes CronJobs, CI schedulers with durable storage, Windows Task Scheduler, and agent harnesses. Do not overlap jobs; the ledger lock rejects a concurrent writer, and stale locks are archived rather than silently deleted.

JSON contract

Success:

{
  "appended_count": 3,
  "duplicate_count": 17,
  "fetched_count": {
    "C0123456789": 16,
    "C0987654321": 4
  },
  "newest_message_checkpoint": {
    "C0123456789": "1700000001.000002",
    "C0987654321": "1700000000.000001"
  },
  "ok": true,
  "run_mode": "incremental",
  "total_ledger_record_count": 303,
  "verification_status": "ok"
}

Failure:

{
  "error_code": "slack_rate_limited",
  "message": "Slack retry budget was exhausted",
  "ok": false,
  "retryable": true,
  "stage": "slack_api"
}

Run results never include message text, author email, filenames, links, attachment content, or tokens.

Data layout

ledger-root/
  channels/<channel-id>/YYYY-MM.jsonl  canonical append-only records
  inbox/<utc-run-id>.jsonl             private run spool, retained only on failure
  locks/stale/                          archived stale-lock evidence
  index.sqlite3                         derived lookup index
  state.json                            derived checkpoints and scan history

Back up canonical JSONL first. The index and state can be rebuilt:

black-lake-slack-ledger rebuild-index --config ./config.toml --ledger-root /var/lib/black-lake-slack-ledger
black-lake-slack-ledger verify --config ./config.toml --ledger-root /var/lib/black-lake-slack-ledger

Never automatically edit, truncate, concatenate, or “repair” canonical JSONL.

Verification

Offline tests use fake Slack responses and temporary ledgers:

python -m unittest discover -s tests -t . -v
python -m compileall -q black_lake_slack_ledger tests
python scripts/audit_public_release.py --history

The repository's evidence and remaining live checks are in docs/verification-report.md. Security and credential guidance is in SECURITY.md.

Known limitation

search.messages is a legacy method and Slack says user search filters can affect its results. The 72-hour overlap reduces transient misses, and weekly full reconciliation enumerates every historical root with replies through channel history, but no polling design can make a stronger claim than the upstream search surface permits. Treat the first approved live backfill and a later full reconciliation as operational validation gates.