Kandi

Best Hack Overall - Hack to the Beat 2026 @ UVA | Devpost

Smart festival glasses that help you protect your hearing, find friends in the crowd, identify songs, and send safety alerts.

Stack

Swift/SwiftUI • CoreBluetooth • AVAudioEngine • ShazamKit • MentraOS • Bun/TypeScript • Express • AudD • Gemini • Shazam API

Setup

MentraOS Backend

cd kandi/MentraOS-Display-Example-App-mainkandi
bun install
cp .env.example .env
bun run dev

Add your MentraOS and recognition API keys to .env:

PACKAGE_NAME=com.wicshackathon.rightsnow
MENTRAOS_API_KEY=your_mentraos_api_key
AUDD_API_KEY=your_audd_key
GEMINI_API_KEY=your_gemini_key
SHAZAM_API_KEY=your_shazam_key
PORT=3000

Expose the backend for the iPhone app:

ngrok http 3000

Set NGROK_URL in .env to the HTTPS forwarding URL.

MentraOS Configuration

Create an app at console.mentra.glass:

• Package: com.wicshackathon.rightsnow
• Server URL: https://your-ngrok-url.ngrok-free.app
• Required capability: microphone/display access for the glasses app

Launch the app on the glasses, then use /api/session/list to confirm an active session.

iOS Companion

Open the Swift app in Xcode:

open "wicsswift copy/festival/Kandi.xcodeproj"

In the app settings, enter the same ngrok HTTPS URL used by the backend. The iPhone companion handles microphone decibel monitoring, BLE friend discovery, ShazamKit recognition, safety keyword detection, and overlay posting.

API

• GET /health - backend health check
• GET /api/session/list - active glasses sessions
• POST /api/overlay/hearing - live dB, risk level, and safe-time overlay
• POST /api/overlay/friends - BLE friend proximity overlay
• POST /api/song/identify - cloud song recognition fallback
• POST /api/friends/safety-alert - broadcast a friend safety alert
• GET /api/friends/has-safety-alert?sessionId=... - poll active safety alerts

How It Works

iPhone mic -> dB + exposure model -> MentraOS backend -> glasses HUD

Friend phones -> BLE RSSI proximity -> iPhone companion -> direction hints on glasses

Concert audio -> ShazamKit/AudD/Gemini/Shazam -> song title on glasses

Safety keyword -> local notification + backend broadcast -> friends' glasses alert

Repo Structure

• kandi/MentraOS-Display-Example-App-mainkandi/ - Bun/TypeScript MentraOS backend
• wicsswift copy/festival/ - SwiftUI iOS companion app
• kandi/DEVPOST.md - Devpost project writeup