Replace Nvidia Jetson by Raspberry Pi as default hardware#70
Draft
urucoder wants to merge 49 commits into
Draft
Conversation
Added configuration for xbee_master program to supervisord.
Added startup logging for local XBee identity and improved error handling.
Added prompts for ROBIN usage and xbee configuration.
Added serial idle reopen feature and improved logging.
Removed rtc_sync_once program configuration from supervisord.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
inference.py writes a sidecar JSON next to each gallery JPEG containing captured_at, image dimensions, and pixel-int bbox per detection. rest_client.py now POSTs once per image to the modern /uploads/image endpoint with metadata JSON instead of N flat-form-field posts per image. SERVER_BASE_URL is standardized to include the /v1 prefix in both envs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… paths docker-compose.yml now binds the XBee FTDI adapter into the container as /dev/xbee_serial via its stable by-id symlink, and mounts /dev/serial so the VE.Direct cable's by-id link is also visible. supervisord.conf points xbee_master_collect.py at /dev/xbee_serial. rest_client.py opens VE.Direct via VE_DIRECT_PORT env var defaulting to the Victron by-id symlink, so it no longer fights the XBee for /dev/ttyUSB0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a new "XBee Radio Config (ROBIN deployments only)" subsection that covers what xbee_configure.py writes to the radio (AP, CE, BR, ID, NI, BD) and that settings persist via WR. Removes the "6 RTC Seeding" step and the "seeds the DS3231 RTC" phrasing from the intro since that flow was deleted from sparrow_setup.sh in ac42811; Wi-Fi and Docker steps renumbered to fill the gap. DS3231 stays in the BOM (still wired up as a kernel RTC via the device-tree overlay).
sparrow/requirements.txt, sparrow/xbee_master_collect.py, and starlink/requirements.txt are restored to the versions actually running on the Pi at 192.168.1.233. sparrow/model_update.py is intentionally kept at the local ONNX-only refactor since the Pi's pre-refactor Triton variant is not the desired version. Also brings in upstream housekeeping that was missing because the repo was initialized fresh rather than cloned: .gitattributes, .github/workflows/codeql.yml, the full .gitignore (replaces the local two-line stub), and documentation/.gitkeep. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
starlink_sleep_schedule.py now reads VE.Direct from the VE_DIRECT_PORT env var (defaulting to the Victron by-id symlink), mirroring the sparrow side fix from 6e71202. docker-compose.yml mounts /dev/serial into the starlink-tools container so the by-id link resolves. Both sparrow.env and starlink.env replace the dead VEDIRECT_PORT/VEDIRECT_BAUD entries with VE_DIRECT_PORT pointing at the Victron cable. The server's /v1/get_schedule, /v1/get_scheduleaudio, and /v1/model_settings endpoints now reject body-only auth with 422 missing X-API-Key. Add the X-API-Key + X-Unit-ID headers to fetch_remote_schedule (sleep_schedule), fetch_settings (audio), and fetch_model_settings (inference). The legacy body-form auth_key/unique_id is left in place for endpoints that still accept it.
Closes alerts for protobuf, Pillow (x5), urllib3 (x2), idna: - Pillow 10.4.0 -> 12.2.0 (PSD/FITS/PDF/font issues, 4 high + 1 medium) - protobuf 4.25.8 -> 5.29.6 (JSON recursion bypass, high) - urllib3 2.6.3 -> 2.7.0 (header leak + decompression bomb, 2 high) - idna 3.7 -> 3.15 (encode bypass, medium) - tritonclient[http]==2.26.0 removed; the runtime is ONNXRuntime-only, tritonclient is not imported anywhere, and the pin was holding protobuf at 4.x. Dropping it unblocks the protobuf 5 bump. Verified end-to-end on the deployed Pi: container rebuild clean, all supervisord processes start, settings POSTs return 200, and an injected test JPEG round-trips through inference -> gallery -> /uploads/image without errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…heck fail
A small systemd-timer-driven bash daemon for SPARROW Pis. Polls
github.com/Clamps251/sparrow-pi every 15 minutes, deploys the newest
git tag matching ^v\d{4}\.\d{2}\.\d{2}, rebuilds the docker-compose
stack, and auto-rolls-back if the new build doesn't pass a 60s
post-restart health check (containers running + no fatal log patterns +
inference.log and restclient_logs.log heartbeating).
Pi-local config (env files, access keys, runtime data, ONNX models,
.git/) is never touched -- preserved via shared --exclude lists on both
the forward apply rsync and the rollback restore rsync.
Updater script lives at updater/sparrow-update.sh; one-time installer
at updater/install.sh writes the systemd unit + timer, the logrotate
config, and prompts for a GitHub PAT (read-only contents) needed while
the repo is private. sparrow_setup.sh prompts at the end of bring-up
to enable the updater on fresh field deployments.
The running script at /usr/local/sbin/ never updates itself; new
versions land via re-running install.sh manually after pulling --
prevents a broken updater commit from ever bricking auto-recovery.
Under `set -euo pipefail`, declarations like
local tag="$1" target="$BACKUP_DIR/$tag"
evaluate every right-hand side before any of the local bindings take
effect, so `$tag` in the second initializer refers to the (unset) outer
scope and bash aborts with "tag: unbound variable". Splitting the
declarations into separate `local` statements in `snapshot_to_backup`
and `restore_backup` fixes the crash mid-deploy, just before the backup
snapshot is taken.
Verified end-to-end on the Pi: v2026.06.09 deploy now runs through
download -> verify -> snapshot -> apply -> build -> recreate -> health
check in ~3 minutes, with current_tag/current_sha written and Pi-local
env + access_key files preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…Hub's
GitHub's /repos/.../tags endpoint returned v2026.06.09 before
v2026.06.09-hotfix1, so jq's `first` would pick the older one and the
updater would always think it was "up to date" once any base date tag
existed. Switching to `sort_by(.name) | last` makes the order
deterministic on the client and gives the natural ordering
v2026.06.09 < v2026.06.09-hotfix1 < v2026.06.10
which is what date-style + hyphen-suffix tags should produce.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…fault The repo is going public, so the PAT prompt during install.sh is unnecessary friction for almost every future install. Drop the interactive prompt and the default config file write. The script still honours an optional `GITHUB_TOKEN` from /etc/sparrow-update.conf when the repo is still private — set it via `sudo env UPDATER_GITHUB_TOKEN=... bash updater/install.sh` and the installer persists it (mode 0600). Existing /etc/sparrow-update.conf files are left in place, so the already-deployed Pi keeps using its current token until the repo flips public and it can be deleted.
…nces - Replace Jetson references with Raspberry Pi 5: BOM brain row, Pi 3 Click Shield description, headless EDID dongle, DC barrel jack, Cat 6 cable label, section header "One-click Raspberry Pi Setup", intro paragraph, and the "Pi OS flash" wording. - Drop Triton-server references: `config.pbtxt` paths removed from the folder-layout block (the script no longer writes them), step 4 retitled "Models" with a note that `Models/tritonserver/` is just a vestigial directory name, Software Dependencies now lists ONNXRuntime (no `tritonclient`, no Triton server), and the `/model_update` endpoint description no longer mentions Triton. - Update the API References block: image POSTs go to `/uploads/image` with bbox-bearing detections JSON; auth now accepts X-API-Key / X-Unit-ID headers alongside the legacy body fields. - New section "🔄 Automatic Updates" covering the tag-driven update flow, what the rsync exclude list preserves, rollback + STUCK behaviour, and how to bootstrap on a fresh Pi. Full operator docs live in updater/README.md.
xbee_master_collect.py now routes incoming files by extension at save time: .mp3 / .wav -> the new --audio-out dir (defaults to /app/recordings in the container, wired up via supervisord.conf), everything else stays in --out (/app/images for the inference pipeline). Robin's MP3 captures flow into the same dir audio.py writes WAV recordings to, so the rest of the audio pipeline is unchanged. rest_client.py picks the Content-Type from the file extension so MP3s ship as audio/mpeg, WAVs as audio/wav, anything else as application/octet-stream. process_and_upload_audio scans for both extensions. No server-side change required -- /audio_uploads already accepts arbitrary audio MIME types; only the upload's Content-Type label changes.
…o.azure.com Three small URL updates: - sparrow.env / starlink.env: SERVER_BASE_URL now points at https://server.sparrowstudio.azure.com/v1 - setup script/sparrow_setup.sh: ONBOARDING_URL matches the same host The deployed Pi keeps its own per-device override (sparrowworld.ddns.net test endpoint) since both .env files are on the auto-updater's preserve list and never get rewritten by a release.
Pairs with f4a8105 (server endpoint rename). The dashboard host moves from dashboard.sparrow-earth.com to www.sparrowstudio.azure.com, and the Terms & Conditions link follows the same host (the /agreement path is preserved). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six Python entry-point scripts hardcode "https://server.sparrow-earth.com" as the fallback for `os.getenv("SERVER_BASE_URL", ...)`. The env files now ship https://server.sparrowstudio.azure.com/v1 by default (see f4a8105), so update the fallback to match — including the /v1 prefix, which the env files have bundled into SERVER_BASE_URL since 945214f. In practice the fallback never fires on a configured Pi (the env file is always present), but stale defaults are misleading when reading the code or running a script standalone for debugging.
sparrow_setup.sh: - REPO_URL now https://github.com/microsoft/SPARROW.git - prompt_github_pat and clone_private_repo removed; replaced with a single clone_repo helper that does a shallow anonymous clone. The main flow drops the prompt_github_pat step. updater/sparrow-update.sh: - REPO_OWNER / REPO_NAME switched to microsoft / SPARROW. - GITHUB_TOKEN kept in as an optional hook for private forks but the comment now flags it as opt-in, not required. Tarball-listing comment and the /repos URL comment updated. updater/README.md: - Bootstrap section says "no prompts, no config" against the public repo. The token subsection is retitled "private fork / auth" and drops the "delete the file once public" instructions since the upstream is now public by default. The Pi at 192.168.1.233 keeps pointing at Clamps251/sparrow-pi via its already-installed /usr/local/sbin/sparrow-update.sh (updaters never self-update). To switch that Pi over, re-run `sudo bash updater/install.sh` manually after this change deploys to its /home/sparrow/Desktop/system/ tree — and /etc/sparrow-update.conf can be deleted since no token is needed anymore.
Assembly & Setup Guide: 6.1 MB -> 10.3 MB (expanded content). Bill of Materials: 3.9 MB -> 216 KB (compressed / trimmed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
torch 2.8.0 -> 2.10.0 (fixes unpack_sequence + lstm_cell CVEs) torchvision 0.23.0 -> 0.25.0 (kept in lockstep for compat) torchaudio 2.8.0 -> 2.10.0 (kept in lockstep for compat) Closes: - #20 (medium) memory corruption via torch.unpack_sequence - #22 (low) memory corruption via torch.lstm_cell Note #21 (low) memory corruption via torch.jit.script has no patched version yet (affects <= 2.12.0). Kept open in Dependabot; unreachable in this codebase because we never JIT-compile user input. Verified end-to-end on the test Pi: rebuild in 88s, container starts clean, all imports succeed, a test JPEG round-trips through inference (letterbox -> ONNX -> torchvision NMS -> PIL save with EXIF) with no errors.
Base images:
- sparrow/Dockerfile: python:3.10-slim-bookworm -> python:3.12-slim-bookworm
- starlink/Dockerfile.starlink: python:3.10-slim -> python:3.12-slim
Cython pre-install (sparrow/Dockerfile): 0.29.36 -> 3.0.11 (0.29 doesn't
build against Python 3.12; keeps the pre-install pattern that pins the
build-env Cython to a known-good version).
Requirements bumps forced by the Python bump (packages without cp312 wheels
at the previous pin):
- pandas 1.4.4 -> 2.2.3
- numba 0.59.1 -> 0.60.0 (audio.py's librosa dep; not imported
directly by SPARROW code)
- llvmlite 0.42.0 -> 0.43.0 (paired with numba)
- onnxruntime 1.18.0 -> 1.20.1 (1.18 predates cp312 wheels)
Verified end-to-end on the test Pi (192.168.1.233):
- Full sparrow rebuild in 110s, starlink in 13s. All wheels resolved
cleanly from PyPI for cp312 aarch64.
- Both containers report Python 3.12.13 and import torch 2.10.0,
torchvision 0.25.0, torchaudio 2.10.0, pandas 2.2.3, numba 0.60.0,
onnxruntime 1.20.1, Pillow 12.2.0 without error.
- Injected test JPEG round-tripped through inference (letterbox -> ONNX
MegaDetector -> torchvision NMS -> Pillow save with EXIF) in ~2s;
gallery + detections.csv row both correct.
- audio.py's pandas usage (DataFrame + read_csv + sort_values + reset_index)
is stable across the 1.4 -> 2.2 jump; no code changes needed.
- No stale tracebacks post-recreate; the ones flagged in the supervisor
err log are historical (May 2026, pre by-id serial fix, cited in
python3.10 stack frames).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Robin retransmits the same JPEG (fresh SID, same bytes) when it doesn't
hear an OK ack in time. The old master saved every one as
<src64>_<sid>_<name>.jpg
so identical content produced a unique filename each time. Downstream:
each duplicate went through inference, landed in gallery, and got
rejected by the server's UNIQUE (project_id, sha256) constraint with a
500. One field unit had 584 files with only 44 distinct sha256s
before we noticed.
New behaviour:
- Compute sha256 of the reassembled payload before saving.
- In-memory LRU seen-set, capped at SEEN_HASHES_MAX=5000.
- If the hash is in the set: log DUPE-SKIP, still send the OK ack so
robin stops retrying, but do not write another copy.
- Startup seeds the set from `--out` (pre-inference images) and
`--audio-out` (raw audio). Does NOT seed from /app/static/gallery:
inference.py re-encodes JPEGs via Pillow when it saves them, so
gallery sha256s don't match what robin retransmits and seeding from
there would produce false negatives (which we confirmed empirically
on the test Pi).
- SAVED log line now includes the sha256 prefix for auditability.
Unique images (different sha256) stay unique. Bit-identical retries
are absorbed silently. Server unique-sha256 500 errors on this device
should drop to zero for genuine mesh retries.
…n't pile up" This reverts commit a228f3b.
|
@urucoder please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.