Orpheus listens to a place and tells you what is there. It runs several machine-learning models over live audio and video — a bird specialist, a corvid specialist, a general sound classifier — and reconciles what they each heard into one picture: one entry per real animal, carrying the evidence that identified it. It runs on hardware you own, and the recordings stay there.
Orpheus listens to a real place, and with playback wired up it makes sound in one. Anywhere you run it is somebody's habitat. Don't deploy it in protected or sensitive areas, don't let it become a nuisance to your neighbors — human or animal — and use playback deliberately: sound directed at wildlife changes behavior, and it carries further than you expect. Running this software means taking that judgment on yourself.
A running station — the dashboard on a live deployment.
The whole collective — capture, all the classifiers, and the correlator — runs in containers against a synthetic or replayed audio source. You do not need a microphone, a camera, or a Jetson to watch a real recording become an identified animal.
You need Docker with Compose v2, Git LFS, and about 10 GB of disk. The last step also
runs on your machine rather than in a container, so it needs Python 3.9 — or uv, which
will fetch it (curl -LsSf https://astral.sh/uv/install.sh | sh). Budget 30 minutes
or so the first time: the model files are a 1.5 GB download, and the classifier images
build PyTorch and ONNX from scratch. A second run takes minutes, off the cache.
git clone https://github.com/scottchronicity/orpheus.git && cd orpheus
git lfs install && git lfs pull # ML models (~1.5 GB)
make sim-up # backplane + correlator + synthetic source
make sim-fleet-up # the real models
make sim-validate # asserts a real clip becomes an entity
make sim-downmake sim-validate is the payoff: it plays a real recording through BirdNET, the corvid
classifier, and the sound classifier, and asserts the correlator turned them into one
entity. The fleet also exposes the dashboard's API on port 8082 — curl localhost:8082/api/config — but it does not serve the dashboard itself, which needs a
built frontend. To click around the UI, run it locally with make dev-stack (see the
quickstarts) or install it on real hardware.
Ready for real hardware? Pick a quickstart below.
The documentation is published at https://scottchronicity.github.io/orpheus/docs/ —
searchable, and the same tree you get by reading docs/ in a checkout. Either
way, start at docs/index.md, which routes you by what you came to do:
| If you want to… | Start here |
|---|---|
| Decide whether this is the right tool | How Orpheus compares |
| Use the dashboard | User Guide |
| Install it yourself | Installation · with an AI assistant |
| Run it safely | Security · Operator's Manual |
| Understand the data | Data models |
| Change how it works | Architecture · Contributing |
| Work on responding agents or vision | "I want to work on responding and vision" |
See What's new for this release, and the CHANGELOG for the history.
Orpheus is a closed loop: an agent that observes the place it is in and acts in it, built on Active Inference — an agent minimizes surprise by predicting its environment and acting to bring those predictions about. Applied to wildlife, that means hearing a crow call, recognizing it as a specific kind of call from a specific individual, and answering it — not from a script, but from a learned model of the interaction.
The observing side is wired up and running on a Jetson Orin NX at a field site: multi-channel audio 24/7 into BirdNET, a custom AVES-based crow vocalization classifier, and a general sound-event model, reconciled into entity-level events and persisted locally. Recognizing animals by sight goes in next as the system moves further into the field, and contributors are welcome to start on it now.
The acting side runs on the same bus: the playback agent is built and the playback request contract is in place. What this repository does not ship is a specific responding agent — the policies that decide what to say, to whom, and when are in active development. Full feedback loops are held to a high bar before they land here, which may keep them out of the public repository for a while. Tools that talk back to animals are worth getting right.
| Stack | Status | Description |
|---|---|---|
| Observe | Running in the field | Audio/video capture, ML inference, event correlation, storage |
| Act | Playback built; responding agents in development | Audio playback, feeder control, interaction policy |
If you are interested in Active Inference, crow cognition, bioacoustics, or edge ML, the responding side is where the interesting work is. See GitHub Discussions to start a conversation.
| Platform | Guide | Time |
|---|---|---|
| macOS (development / demo) | macOS Quick Start | ~15 min |
| Jetson Orin NX (production) | Jetson Quick Start | ~30 min |
| Windows (WSL2, untested) | Windows Quick Start | ~20 min* |
| Linux (dev/demo, untested) | Linux Quick Start | ~15 min* |
All guides are self-contained — pick the one for your hardware and go.
* The Windows and Linux guides have not yet been verified end-to-end. They're published to give future contributors a starting point — expect rough edges and please open an issue or PR if you hit one.
Our working philosophy — protect the live field station, earn your domain, hold opinions loosely — is in CONTRIBUTING, and the reasoning behind specific architectural choices lives in the ADRs.
Our roadmap lives in docs/backlog.json, which seeds the labels, milestones, and issues on GitHub Issues. It holds outstanding work only — 44 open stories, grouped into eight themes, each theme a milestone:
| Theme | What it covers | Open items |
|---|---|---|
| Detection & Identity | Recognizing what is out there and naming it consistently across classifiers | 6 |
| Actuation & Response | Playing sound back into the world, safely, and acting on what the system has learned | 4 |
| Event Bus | The transport between agents and the durable stream of what happened | 3 |
| Dashboard & Interfaces | What a person looks at, filters, and asks questions through | 7 |
| Observability | Knowing what the system itself is doing, and what the weather was while it did it | 3 |
| Data & Sharing | Getting recordings and detections out of the station, deliberately and privately | 5 |
| Infrastructure & Testing | Build, deploy, configure, simulate, and prove the thing still works | 10 |
| Edge Hardware | Surviving a sealed box outdoors on a small computer with a finite disk | 6 |
The counts are open stories, not progress — a short list can mean a theme is nearly done or that it has barely been scoped.
Browse GitHub Issues for live tracking. Filter by a milestone for a theme's open work, or by good first issue for something small and self-contained to start on.
flowchart TD
subgraph Sensors["Sensors (Edge Hardware)"]
MIC["Microphones\n(4-channel USB)"]
CAM["IP Cameras\n(RTSP)"]
GPSDEV["GPS Dongle\n(USB Serial)"]
end
subgraph Observe["Observe Stack"]
AM["orpheus-agent-audio-motion\nCapture & motion detection"]
VM["orpheus-agent-video-motion\nMotion detection & clip recording"]
SNAP["orpheus-agent-video-snapshotter\nPeriodic JPEG snapshots"]
GPS["orpheus-gps\nLocation tracking"]
BD["orpheus-agent-bird-detection\nBirdNET ONNX inference"]
AE["orpheus-agent-audio-events\nPANNs AudioSet sound classifier"]
CD["orpheus-agent-crow-detection\nAVES embedder + classifier\n(runs on corvid signals)"]
EC["orpheus-agent-event-correlator\nTemporal fusion → EntityEvents"]
end
subgraph Bus["Messaging Backplane (orpheus-backplane)"]
BUS[("NATS + JetStream\nlocalhost:4222 (mqtt fallback)")]
end
subgraph Storage["Storage"]
FLAC["FLAC audio clips\n/data/orpheus/audio/"]
MP4["MP4 video clips\n/data/orpheus/video/"]
JPG["JPEG snapshots\n/data/orpheus/video/snapshots/"]
DB["DetectionDB\n(SQLite)"]
end
subgraph Act["Act Stack"]
PB["orpheus-agent-audio-playback\nPlayback engine (built)"]
POLICY["Interaction Policy\n(in development — not shipped here yet)"]
end
MIC --> AM
CAM --> VM
CAM --> SNAP
GPSDEV --> GPS
AM -->|"orpheus/audio/motion/events"| BUS
VM -->|"orpheus/video/motion/events"| BUS
GPS -->|"orpheus/state/location"| BUS
BUS -->|"orpheus/audio/motion/events"| BD
BUS -->|"orpheus/audio/motion/events"| AE
BD -->|"orpheus/detection/bird/events"| BUS
AE -->|"orpheus/detection/audio/events"| BUS
BUS -->|"corvid signals from either classifier"| CD
CD -->|"orpheus/detection/crow/events"| BUS
BUS -->|"detection events"| EC
EC -->|"orpheus/entities/animal"| BUS
EC --> DB
AM --> FLAC
VM --> MP4
SNAP --> JPG
BUS -->|"orpheus/entities/animal"| POLICY
POLICY -->|"orpheus/audio/playback/request"| BUS
BUS -->|"orpheus/audio/playback/request"| PB
All inter-agent communication flows through the local messaging backplane (orpheus-backplane: NATS + JetStream by default, mosquitto as the one-line fallback — the orpheus/... topic hierarchy is the wire contract on either backend). No agent talks directly to another, so a new analysis agent needs no changes to any existing agent: it subscribes to the topics it cares about and publishes its results. Wiring it into the build, CI, systemd and the manifest generator is a checklist — see Adding an agent.
The full topic hierarchy:
| Topic Prefix | Purpose |
|---|---|
orpheus/audio/motion/events |
Raw audio motion triggers from microphones |
orpheus/video/motion/events |
Raw video motion triggers from cameras |
orpheus/detection/bird/events |
BirdNET species identifications |
orpheus/detection/crow/events |
Crow vocalization behavior classifications |
orpheus/detection/audio/events |
General sound-event classifications (PANNs / AudioSet) |
orpheus/entities/animal |
Correlated entity events (the "animal was here" record) |
orpheus/audio/playback/request |
Commands sent to the playback agent |
orpheus/state/location |
GPS position (retained) |
orpheus/system/{agent}/health |
Per-agent health and liveness |
orpheus/
├── platform/
│ ├── jetson-orin-nx-yahboom/ # Hardware-specific: ALSA config, GPIO, display, networking
│ └── orpheus-common/ # Shared Python library (config, event bus, logging, storage, DetectionDB)
│
├── agents/ # Hardware-agnostic detection and analysis agents
│ ├── orpheus-agent-audio-motion/ # Layer 1: Audio energy detection, FLAC recording
│ ├── orpheus-agent-audio-events/ # Layer 2: PANNs AudioSet general sound classifier
│ ├── orpheus-agent-bird-detection/ # Layer 2: BirdNET ONNX species ID
│ ├── orpheus-agent-crow-detection/ # Layer 2: AVES + multi-task crow classifier
│ ├── orpheus-agent-event-correlator/ # Layer 3: Temporal fusion → EntityEvents
│ ├── orpheus-agent-video-motion/ # Layer 1: Camera motion detection, MP4 recording
│ ├── orpheus-agent-video-snapshotter/# Layer 1: Periodic JPEG snapshots
│ ├── orpheus-agent-video-timelapser/ # Layer 3: Timelapse generation from snapshots
│ └── orpheus-agent-audio-playback/ # Act: Plays audio files on a bus request
│
├── services/ # Hardware-agnostic infrastructure services
│ ├── orpheus-backplane/ # Messaging backplane (NATS+JetStream; mqtt fallback)
│ ├── orpheus_ui/ # Web UI (FastAPI + React + TypeScript)
│ ├── orpheus-gps/ # GPS location service
│ └── orpheus-bluetooth-autoconnect/ # Bluetooth speaker auto-connection
│
├── make/ # Shared Makefile includes (deploy, python, lint, service)
├── hardware/ # Hardware abstraction layer (audio/video device wrappers)
├── artifacts/ # ML models, recordings, calibration data (Git LFS)
├── tools/ # Development and deployment utilities
├── docs/ # Architecture docs, ADRs, component instructions
└── tests/ # Integration and infrastructure tests
The agents/ and services/ directories are completely hardware-agnostic. They contain no Jetson-specific code. All hardware-specific configuration (ALSA device aliases, GPIO mappings, display setup, network configuration) lives exclusively in platform/jetson-orin-nx-yahboom/.
The shared library at platform/orpheus-common/ provides the common abstractions — configuration, the event-bus client, logging, storage path helpers, and the DetectionDB — that every agent and service depends on. It too is hardware-agnostic.
This separation means that porting Orpheus to a new single-board computer (e.g., a Raspberry Pi) requires only adding a platform/raspberry-pi/ directory. The agents themselves do not change. See CONTRIBUTING.md for the porting strategy.
PreRollRingBuffer[T]— the ring buffer shared by the audio and video pipelines, sized in seconds rather than raw capacity.ClusterManager— groups raw detections into entity-level events.ChannelProcessor— the per-microphone pipeline composing the two.
All pull requests must pass:
make test # pytest (≥70% coverage enforced by CI)
make lint # Ruff linting (zero errors)Run these locally before pushing. Cloud CI is expensive.
GitHub Actions runs path-filtered tests on every PR — only the components affected by the diff are tested, except when platform/orpheus-common/ changes (which triggers all tests).
Each component has a dedicated test job. The ci-complete gate job is the only required status check in branch protection.
Adding a component to CI touches six places — see CI/CD.
We welcome contributions at every level — from fixing typos to building the Active Inference policy layer.
Good first issues: small, self-contained, and genuinely scoped — see the good first issue label.
Meaty problems: visual recognition, the interaction policy, the read-only mirror and public portal, contributing recordings back to the models we use.
Please read CONTRIBUTING.md — especially the Python 3.9 constraint — before opening a PR.
For architectural questions and "I want to build X" discussions, use GitHub Discussions.
ML models, audio samples, and calibration data are stored in Git LFS. After cloning:
git lfs install
git lfs pullPlease report security vulnerabilities responsibly. See SECURITY.md for our policy.
MIT — see LICENSE.
Orpheus depends on the work of the bioacoustics and machine-learning research communities:
- BirdNET: All bird species classification is powered by the BirdNET-Analyzer by the Cornell Lab of Ornithology and Chemnitz University of Technology.
- AVES: Our crow vocalization embedding pipeline uses the AVES (A Bioacoustic Transformer) foundation model.
- NVIDIA: Deep gratitude to the Jetson team for the hardware and JetPack SDK that makes 24/7 on-device inference possible.
- Community: Third-party audio samples used for testing and calibration are credited in artifacts/audio-samples/README.md.
| Document | Purpose |
|---|---|
| docs/ | The documentation site — start at docs/index.md |
| AGENTS.md | Entry point for AI coding agents (non-negotiables + navigation map) |
| docs/ARCHITECTURE.md | Detailed system architecture |
| docs/adr/ | Architectural Decision Records |
| CONTRIBUTING.md | Contribution guidelines |
| CHANGELOG.md | Release history |