Autonomous infrastructure guardian for the Casper Network — powered by a local Phi-3 LLM, verified by x402 on-chain micro-payments, and visualised in real-time through Grafana.
CSPR-Sentinel is a self-healing infrastructure agent that watches your Casper validator node, reasons about disk crises with an air-gapped local AI, and — after on-chain payment confirmation — autonomously provisions a failover node, all without a human in the loop.
Built for the Casper Hackathon 2026.
| Problem | Traditional approach | CSPR-Sentinel |
|---|---|---|
| Disk crisis on validator | PagerDuty alert → human wakes up → manual action | Local LLM decides → x402 payment → node-b live in seconds |
| AI reasoning | Cloud API (latency, privacy, cost) | Phi-3 runs fully local via Ollama — no data leaves the machine |
| Payment verification | Off-chain webhook / trust-based | x402 on-chain Casper deploy — immutable audit trail |
| Grafana visibility | Real disk only (no simulation) | textfile_collector feeds mock metrics so the crisis spike is visible without touching the disk |
HOST MACHINE
============
daemon/main.py
--------------
psutil / .force_crisis
|
| disk > 90%
v
+--------+ POST /api/generate +------------------+
| IDLE | ---------------------> | Ollama (Phi-3) |
+---+----+ +--------+---------+
| <-- TRIGGER_FAILOVER ---------------+
|
| trigger_failover_payment()
| writes daemon/.trigger_signal
v
+---------+
| PENDING | (Ollama NOT called)
+----+----+
|
| .trigger_signal consumed by gateway
v
+----------+
| MIGRATED | (Ollama NOT called, .force_crisis removed)
+----------+
|
v
daemon/textfile_collector/cspr_sentinel.prom (written every 10s)
|
v
[node-exporter :9100] --> [Prometheus :9090] --> [Grafana :3000]
gateway/mock_cloud_provider.py
------------------------------
polls daemon/.trigger_signal every 3s
|
| file detected
v
[EVENT] x402 Payment Received on Casper Network!
|
| 2s network confirmation simulation
v
docker compose --profile failover up -d casper-node-b
|
| success
v
deletes .trigger_signal --> daemon: PENDING -> MIGRATED
DOCKER NETWORK: cspr-sentinel-net
casper-node-a :8080 primary (always up)
casper-node-b :8081 failover (profile=failover, started on demand)
prometheus :9090
grafana :3000
node-exporter :9100 + textfile_collector/ volume mount
daemon start
|
v
+--------+ <------------------------------------------------+
| IDLE | disk <= 90%: log "healthy", sleep 10s |
+---+----+ |
| |
| disk > 90% |
v |
Phi-3 consulted via Ollama |
| |
+-- response: NO_ACTION --------------------------------+ (loop)
|
| response: TRIGGER_FAILOVER
v
trigger_failover_payment()
writes daemon/.trigger_signal
|
v
+---------------------------+
| PENDING | Ollama NOT called
| polls signal file | logs "waiting for gateway..."
+----------+----------------+
|
| daemon/.trigger_signal deleted by gateway
v
+-----------------------------+
| MIGRATED | Ollama NOT called
| .force_crisis self-removed | logs "node-b is live"
| node-b is serving traffic | stays here until daemon restart
+-----------------------------+
| Path | Role |
|---|---|
daemon/main.py |
Core sentinel loop — disk polling, state machine, Ollama client, textfile metrics writer |
daemon/mcp_server.py |
System metrics (psutil) + trigger_failover_payment() (x402 mock + signal file) |
gateway/mock_cloud_provider.py |
x402 event listener — detects signal, runs docker compose up casper-node-b |
contracts/odra_registry/ |
Rust/Odra smart contract — on-chain NodeRegistry (validator pubkey → IP) |
gateway/prometheus.yml |
Prometheus scrape config (node-exporter + daemon textfile metrics) |
simulate_crisis.sh |
One-shot crisis injector — writes .force_crisis to trigger the full pipeline |
daemon/textfile_collector/ |
Prometheus textfile dir — daemon writes cspr_sentinel.prom every 10s |
- Docker Desktop (with Compose v2)
- Python 3.11+
- Rust + nightly toolchain (
rustup toolchain install nightly) - Ollama with
phi3model pulled
# Pull the Phi-3 model (one-time, ~2 GB)
ollama pull phi3git clone https://github.com/<your-org>/cspr-sentinel.git
cd cspr-sentinel
python3 -m venv .venv
.venv/bin/pip install -r daemon/requirements.txtdocker compose up -d casper-node-a prometheus grafana node-exporter| Service | URL | Credentials |
|---|---|---|
| Grafana | http://localhost:3000 | admin / sentinel123 |
| Prometheus | http://localhost:9090 | — |
| casper-node-a (primary) | http://localhost:8080/health | — |
Open three terminal tabs in the project root, then execute in order:
cd /path/to/cspr-sentinel
.venv/bin/python3 gateway/mock_cloud_provider.pyExpected idle output:
2026-06-28 19:40:00 [INFO] Mock Cloud Gateway started — watching .../daemon/.trigger_signal (poll every 3s)
cd /path/to/cspr-sentinel
.venv/bin/python3 daemon/main.pyExpected idle output:
2026-06-28 19:40:01 [INFO] CSPR-Sentinel daemon started threshold=90% interval=10s state=IDLE
2026-06-28 19:40:01 [INFO] disk=50.0% sync_lag=False used=13.8GB/228.3GB state=IDLE
2026-06-28 19:40:01 [INFO] [sentinel] System healthy — no action
cd /path/to/cspr-sentinel
./simulate_crisis.shThen watch Terminals 1 and 2 for the autonomous response.
The following is captured output from a successful end-to-end run:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ DAEMON (Terminal 2) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2026-06-28 19:42:15 [INFO] CSPR-Sentinel daemon started threshold=90% interval=10s state=IDLE
2026-06-28 19:42:15 [INFO] disk=50.0% sync_lag=False used=13.8GB/228.3GB state=IDLE
2026-06-28 19:42:15 [INFO] [sentinel] System healthy — no action
# ./simulate_crisis.sh is run in Terminal 3
2026-06-28 19:42:25 [WARNING] [MOCK CRISIS] real=50.0% → mocked=93.0% state=IDLE
2026-06-28 19:42:25 [WARNING] [sentinel] Disk 93.0% exceeds threshold — consulting Phi-3
2026-06-28 19:42:28 [INFO] [ollama] raw response: {"action": "TRIGGER_FAILOVER", "reason": "Disk space is critically full"}
2026-06-28 19:42:28 [WARNING] [sentinel] Phi-3 → TRIGGER_FAILOVER (Disk space is critically full)
2026-06-28 19:42:28 [INFO] [x402] On-chain payment process started for new node provisioning...
2026-06-28 19:42:28 [INFO] [x402] Casper deploy hash: 0x3f9a1d7c2b84e605a1f3c9d2e7b04f8a1c6d3e9f2a5b8c1d4e7f0a3b6c9d2e5
2026-06-28 19:42:28 [INFO] [sentinel] Trigger signal written to daemon/.trigger_signal
2026-06-28 19:42:28 [INFO] [sentinel] State → PENDING (gateway will provision node-b)
2026-06-28 19:42:38 [WARNING] [MOCK CRISIS] real=50.0% → mocked=93.0% state=PENDING
2026-06-28 19:42:38 [INFO] [sentinel] PENDING — waiting for gateway to finish provisioning node-b...
# (Phi-3 is NOT consulted again — state machine blocks redundant calls)
2026-06-28 19:42:45 [INFO] [sentinel] State → MIGRATED (casper-node-b is live)
2026-06-28 19:42:45 [INFO] [sentinel] .force_crisis removed — mock crisis cleared
2026-06-28 19:42:55 [INFO] [sentinel] MIGRATED — casper-node-b serving traffic. System stable. Restart daemon to reset.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━ GATEWAY (Terminal 1) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2026-06-28 19:42:28 [INFO] [gateway] Signal file detected: daemon/.trigger_signal
2026-06-28 19:42:28 [INFO] [gateway] Signal contents:
FAILOVER_TRIGGERED
deploy_hash=0x3f9a1d7c2b84e605a1f3c9d2e7b04f8a1c6d3e9f2a5b8c1d4e7f0a3b6c9d2e5
timestamp=1751140948.0
2026-06-28 19:42:28 [INFO] [EVENT] x402 Payment Received on Casper Network! Processing provisioning...
2026-06-28 19:42:28 [INFO] [gateway] Simulating network confirmation — waiting 2s...
2026-06-28 19:42:30 [INFO] [gateway] Running: docker compose --profile failover up -d casper-node-b
2026-06-28 19:42:33 [INFO] [gateway] docker compose stdout: Container cspr-node-b Started
2026-06-28 19:42:33 [INFO] [gateway] Trigger signal file removed.
2026-06-28 19:42:33 [INFO] [SUCCESS] Phi-3 Approved the Failover. x402 Payment Verified.
casper-node-b is now LIVE and serving traffic!
Add the Prometheus datasource (auto-provisioned) and create panels using the queries below.
cspr_sentinel_disk_usage_percent{node="cspr-node-a"}
Grafana panel showing stable disk usage at ~50% before crisis injection. (assets/baseline_metrics.png — add screenshot after running the demo)
cspr_sentinel_disk_usage_percent{node="cspr-node-a"}
cspr_sentinel_mock_crisis_active{node="cspr-node-a"}
cspr_sentinel_failover_state{node="cspr-node-a"}
The moment ./simulate_crisis.sh runs: disk gauge spikes from 50% → 93%, mock_crisis_active flips to 1, then failover_state transitions 0 → 1 → 2 as the state machine progresses.
cspr_sentinel_failover_state{node="cspr-node-a"}
Time-series showing the step function: 0 (IDLE) → 1 (PENDING) → 2 (MIGRATED) after docker compose completes. (assets/state_machine_timeline.png — add screenshot after running the demo)
The contracts/odra_registry crate contains a production-ready Casper smart contract built with the Odra framework.
#[odra::module(events = [AddressUpdated])]
pub struct NodeRegistry {
nodes: Mapping<String, String>, // validator_pubkey → ip_address
}Entry points:
| Function | Args | Description |
|---|---|---|
set_node_address |
public_key: String, ip: String |
Register or update a node's IP; emits AddressUpdated event |
get_node_address |
public_key: String |
Returns the registered IP or empty string |
Build:
cd contracts/odra_registry
cargo +nightly check # type-check
cargo +nightly test # run unit tests (odra-vm backend)trigger_failover_payment()
│
├── logs "[x402] On-chain payment process started..."
├── generates fake_deploy_hash = 0x<32 random bytes>
├── logs "[x402] Casper deploy hash: 0x..."
└── writes daemon/.trigger_signal
│
└──▶ gateway polls and detects this file
│
└──▶ docker compose up casper-node-b
│
└──▶ deletes .trigger_signal
│
└──▶ daemon: PENDING → MIGRATED
In production, trigger_failover_payment() would call the Casper RPC to deploy the NodeRegistry contract entry point and stream the deploy hash back as confirmation.
# Remove crisis artifacts
rm -f daemon/.force_crisis daemon/.trigger_signal
# Stop failover node
docker compose --profile failover stop casper-node-b
# Restart daemon (resets state machine to IDLE)
# Ctrl+C daemon process, then re-run:
.venv/bin/python3 daemon/main.pycspr-sentinel/
├── daemon/
│ ├── main.py # State machine + Ollama client + textfile writer
│ ├── mcp_server.py # psutil metrics + x402 trigger
│ ├── requirements.txt # psutil, requests, mcp
│ └── textfile_collector/ # Prometheus textfile dir (written by daemon)
│ └── cspr_sentinel.prom # auto-generated each cycle
├── gateway/
│ ├── mock_cloud_provider.py # x402 event listener + docker compose trigger
│ ├── prometheus.yml # Scrape config
│ ├── nginx-a.conf # Primary node config
│ └── nginx-b.conf # Failover node config
├── contracts/
│ └── odra_registry/ # Rust/Odra smart contract
│ ├── Cargo.toml
│ ├── rust-toolchain.toml # nightly channel
│ └── src/lib.rs # NodeRegistry contract + tests
├── grafana/
│ └── provisioning/
│ └── datasources/
│ └── prometheus.yml # Auto-provision Prometheus datasource
├── assets/ # Screenshots for this README
├── simulate_crisis.sh # One-shot crisis injector
└── docker-compose.yml # Full stack definition
| Layer | Technology |
|---|---|
| AI Reasoning | Ollama + Phi-3 (local, air-gapped) |
| On-Chain Payment | x402 protocol on Casper Network (mock deploy in prototype) |
| Smart Contract | Odra Framework 2.8.2 (Rust, nightly) |
| Metrics | Prometheus + node-exporter + textfile collector |
| Visualisation | Grafana 10+ |
| Containerisation | Docker Compose v2 |
| System Agent | Python 3.11 + psutil |
Built with ❤️ for the Casper Hackathon 2026.
"The best infrastructure is the kind that fixes itself."