pfc-archiver-influxdb

Autonomous cold-storage archive daemon for InfluxDB 2.x — compresses old time-series data to pfc-jsonl format and moves it to local storage or S3.

The problem

InfluxDB doesn't have a native cold-storage tier. Old measurements stay in your hot bucket, consuming disk and memory. Manual exports are error-prone and don't scale. Deleting without archiving means data loss.

What it does

Runs as a daemon (or one-shot via --once). On each cycle:

1. Scan — finds records older than retention_days in the configured measurement
2. Export — Flux query → JSONL (long format, one line per field-value record)
3. Compress — pfc_jsonl compress → .pfc + .pfc.bidx (random-access index)
4. Upload — copies archive to local path or s3://bucket/prefix/
5. Verify — decompresses and counts rows; must match exported count
6. Delete — removes archived records from InfluxDB via the Delete API (only if delete_after_archive = true)
7. Log — appends a JSON run entry to archive_runs.jsonl

Requirements

• Python 3.8+
• pfc_jsonl binary on PATH or via --pfc-binary:

# Linux x64:
curl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-linux-x64 \
     -o /usr/local/bin/pfc_jsonl && chmod +x /usr/local/bin/pfc_jsonl

# macOS Apple Silicon (M1–M4):
curl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-macos-arm64 \
     -o /usr/local/bin/pfc_jsonl && chmod +x /usr/local/bin/pfc_jsonl

# macOS Intel:
curl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-macos-x64 \
     -o /usr/local/bin/pfc_jsonl && chmod +x /usr/local/bin/pfc_jsonl

# Windows (PowerShell):
Invoke-WebRequest https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-windows-x64.exe `
  -OutFile "$env:LOCALAPPDATA\Microsoft\WindowsApps\pfc_jsonl.exe"

• InfluxDB 2.x (OSS or Cloud)

pip install pfc-archiver-influxdb
# or
pip install influxdb-client tomli  # Python < 3.11

Quick start

# 1. Copy and edit config
cp config/influxdb.toml my-influxdb.toml
# edit: url, token, org, bucket, measurement, retention_days, output_dir

# 2. Dry-run first (no data moved or deleted)
pfc-archiver-influxdb --config my-influxdb.toml --dry-run

# 3. Single cycle
pfc-archiver-influxdb --config my-influxdb.toml --once

# 4. Daemon mode (runs every interval_seconds)
pfc-archiver-influxdb --config my-influxdb.toml

Configuration

[db]
url         = "http://localhost:8086"
token       = "my-token"
org         = "my-org"
bucket      = "my-bucket"
measurement = "logs"
# scan_start = "2022-01-01T00:00:00Z"  # optional: speeds up scan on large buckets

[archive]
retention_days       = 90        # archive data older than this
partition_days       = 30        # one .pfc file per partition window
output_dir           = "./local-archives/"  # or "s3://bucket/prefix/"
verify               = true
delete_after_archive = false     # set true only after validating archives
log_dir              = "./archive_logs/"

[daemon]
interval_seconds = 3600

Export format

Long format — one JSONL line per field-value record:

{"timestamp": "2024-01-01T00:00:00Z", "_measurement": "cpu", "_field": "usage_idle", "_value": 98.2, "host": "web-01", "region": "eu-west-1"}
{"timestamp": "2024-01-01T00:00:00Z", "_measurement": "cpu", "_field": "usage_user", "_value": 1.5, "host": "web-01", "region": "eu-west-1"}

Tags are preserved as extra columns. _start / _stop (InfluxDB metadata) are excluded.

Querying archives with pfc-duckdb

-- Install: https://github.com/ImpossibleForge/pfc-duckdb
LOAD pfc;
SELECT timestamp, _field, _value, host
FROM pfc_scan('/archives/cpu__20240101__20240201.pfc')
WHERE _field = 'usage_idle'
  AND timestamp >= '2024-01-10' AND timestamp < '2024-01-20';

Related

• pfc-jsonl — compression engine (random access, timestamp index)
• pfc-duckdb — DuckDB extension for querying .pfc files
• pfc-archiver-clickhouse — same pattern for ClickHouse
• pfc-archiver-cratedb — same pattern for CrateDB
• pfc-archiver-questdb — same pattern for QuestDB

Disclaimer

pfc-archiver-influxdb is an independent open-source project and is not affiliated with, endorsed by, or associated with InfluxData, Inc. or the InfluxDB project.

---

License

pfc-archiver-influxdb (this repository) is released under the MIT License — see LICENSE.

The PFC-JSONL binary (pfc_jsonl) is proprietary software — free for personal and open-source use. Commercial use requires a license: info@impossibleforge.com