devopsphone

Modular multi-region monitoring agent. Core logic is Rust; checks and notifiers are drop-in plugins.

Project layout

devopsphone/
├── Cargo.toml
├── config/
│   ├── devopsphone.toml      # node settings
│   └── cluster.example.toml  # mesh + quorum + notify_nodes
├── plugins/
│   ├── checks/               # *.toml check manifests
│   ├── notifiers/            # *.toml notifier manifests
│   ├── scripts/              # plugin executables
│   └── ansible/              # VM playbooks + inventory
└── src/
    ├── main.rs               # thin binary entry
    ├── lib.rs
    ├── config.rs
    ├── scheduler.rs
    ├── alert_state.rs
    ├── plugin_types.rs
    ├── plugin_loader.rs
    ├── script_runner.rs
    ├── legacy/               # optional built-in http/vm checks
    └── mesh/                 # neighbor sync, quorum, election
        ├── config.rs
        ├── store.rs
        ├── client.rs
        ├── server.rs
        ├── quorum.rs
        ├── election.rs       # elected notifier leader
        └── runtime.rs

Quick start

cp .env.example .env
cargo build
set -a && source .env && set +a
RUST_LOG=devopsphone=info cargo run

Enable Telegram: plugins/notifiers/telegram_stub.toml → enabled = true.

Plugins

Add a check: plugins/checks/my_check.toml + plugins/scripts/my_check.sh
Add a notifier: plugins/notifiers/my_notifier.toml + plugins/scripts/my_notifier.sh

See src/plugin_types.rs for the JSON contract.

Mesh + elected notifier

When [cluster] enabled = true:

1. Nodes sync observations over HTTP (src/mesh/)
2. Quorum engine decides warning vs critical vs observer-offline
3. Only one node sends notifications per incident — deterministic leader election (src/mesh/election.rs)

Configure notifier candidates in config/cluster.toml:

notify_nodes = ["devopsphone-armenia-1", "germany-1", "china-1"]

All nodes compute the same leader from incident_id + alive notify_nodes. Non-leaders log skipping notify (not elected leader).

If notify_nodes is omitted, defaults to this node + all neighbors.

Config

File	Purpose
config/devopsphone.toml	Node id, scheduler, plugin paths
config/cluster.toml	Mesh listen addr, neighbors, policies, notify_nodes
.env	TELEGRAM_*, DEVOPSPHONE_SSH_KEY, DEVOPSPHONE_MESH_SECRET

Tests

cargo test