Tally

A React Native app that counts faces in the camera frame, live. There is no shutter: a WIDER FACE-trained YOLO26 runs on TFLite on-device as fast as the phone allows, corner brackets follow the faces between runs, and the count updates as you move the camera. A scan writes itself to history once the number holds still.

Counting is fully on-device and works with no signal — no frame leaves the phone. Naming a face does not: tap a bracket and the crop goes to a face service you host yourself, which owns the embeddings, the threshold and the decision. The app never sees a vector.

The bundled detector is an Ultralytics YOLO26 export, licensed AGPL-3.0. Review the terms (or obtain a commercial licence) before shipping a closed-source build.

Features

• Continuous detection, no shutter. A worklet runs the model on the camera thread whenever the previous pass has finished, rests for max(200ms, last pass cost), and reports back to JS. Measured on a Tecno LI6: 176–185 ms per pass, 2.5 passes/second, preview holding ~16.5 fps with nothing rendering slower than 26 ms.
• Back-pressure, not just throttling. The worklet refuses to detect while a result is still queued for JS. Without it the two threads decouple, the backlog lands in one burst, and React ends it with "maximum update depth exceeded".
• Face tracking by overlap. trackFaces matches each round's detections to the faces already on screen (greedy, best overlap first), so a face keeps one stable id while it moves and survives up to TRACK_MAX_MISSED missed rounds. It also returns the previous state object unchanged when nothing moved — a new array every round re-rendered the whole overlay at 4Hz and read on screen as a flicker.
• Scans record themselves. When the count stops changing for 2 seconds (STABLE_MS), the scene is written to history with a snapshot. Keyed on the count, so holding the camera on the same people records once rather than every two seconds.
• YOLO26 trained on WIDER FACE, single class — [1, 3, 320, 320] float32 NCHW input, raw [1, 5, 2100] head (4 box coordinates + 1 score) with no NMS in the graph. Shapes read straight out of the FlatBuffer, not guessed — see assets/models/README.md.
• Server-side recognition. MediaPipe FaceMesh (468 landmarks, on-device) gates on head pose and supplies the five alignment points; the server decides who a face belongs to (POST /v1/face/verify). A face turned too far never leaves the phone. Without ARCFACE_API_URL configured, counting still works and nobody is named.
• Two crops per face, for two different models. A tight 224px crop for the embedding, and a 2.7× context crop at 160px for liveness — MiniFASNet reads the screen bezel, the paper edge and the reflections around a face, so a tighter window leaves it answering confidently about nothing.
• Account-gated with Supabase, or skip it. AuthScreen is the only thing rendered until useAuth reports a real session — or until you continue as a guest, in which case the camera works and every history write, local and cloud, is disabled.
• Uploads survive being offline. Counting works with no signal, so an upload that fails cannot just be logged and forgotten. Every scan is queued in syncQueue.ts before the request goes out and cleared only once it lands; the queue is retried on launch and whenever the app returns to the foreground. Deletes queue the same way, so a scan removed offline does not come back on the next restore.
• Older scans paged back from the cloud. Local history is capped at HISTORY_LIMIT (50) but the Supabase table is not. A "show older" button pulls PAGE (25) rows down on demand — a button rather than infinite scroll, because each row downloads a thumbnail. Paged-in records are display state only and never written back to MMKV.
• A rolling week summary, CSV export of everything or of a ticked selection, haptic alert on record, instant runtime language switching (Vietnamese is the source of truth, English typed against it), and an adaptive portrait/landscape layout.
• The app draws its own dialogs. Dialog.tsx replaces React Native's Alert, which renders the OS dialog — Material on Android, UIKit on iOS — so the surface asking to delete a scan looked nothing like the screen it was asked from. It is an absolutely positioned overlay rather than a Modal, because stacking Modals on Android means two windows fighting over the back button.
• Architecture enforced by lint, not by convention. shared and core may not import a feature, and no feature may reach past another's index.ts. See docs/architecture/.

Requirements

• Node.js >= 22.11.0
• React Native 0.86.2, bare workflow (no Expo), New Architecture
• Android: minSdkVersion 26, compileSdk/targetSdk 36
• iOS: React Native's min_ios_version_supported, plus CocoaPods via Bundler
• A physical device with a camera — this pipeline does not run on simulators
• A network connection for sign-in, history sync and recognition (detection itself works offline)

Installation

git clone https://github.com/HienNguyen0205/Tally.git

npm install

postinstall runs patch-package automatically to apply the react-native-fast-tflite patch: in release builds the .tflite asset is bundled into the APK, so Image.resolveAssetSource() returns a resource name rather than a URL, making URL(path) throw MalformedURLException and the model fail to load. The patch resolves it as a raw/drawable resource first.

Copy the environment example and fill in your own values:

cp .env.example .env

Variable	Purpose
SUPABASE_URL	Supabase project URL — auth and history sync
SUPABASE_ANON_KEY	Supabase anon key
ARCFACE_API_URL	Base URL of the face service, no path. Empty or absent disables recognition, which is a supported state
DETECT_THRESHOLD	How confident the detector must be before a face is shown, 0–1. Absent or unparseable falls back to 0.5

These are read at build time via react-native-dotenv (@env, see env.d.ts). .env is gitignored, and the values are inlined at bundle time — restart Metro with --reset-cache after changing it.

There is no API token for the face service. Its endpoints authenticate with the signed-in user's Supabase JWT, because they need to know who is asking; an app-wide secret baked into the bundle could only ever say which app.

iOS needs the Pods step as well:

bundle install

bundle exec pod install

Running

Start Metro:

npm start

Then build and run from another terminal:

npm run android

npm run ios

All scripts

Script	What it does
npm start	Metro bundler
npm run android / npm run ios	build, install, run
npm run lint	ESLint — including the architecture rules
npm run typecheck	tsc --noEmit
npm test	Jest
npm run verify	lint + typecheck + test — what CI runs
npm run release	signed APKs, one per phone ABI
npm run release:aab	App Bundle for the Play Store
npm run release:github	build, then publish a GitHub Release

The three release* scripts wrap fastlane lanes, so they need bundle install first. Publishing normally happens by pushing a tag rather than by running release:github by hand.

How it works

Detection runs in a worklet on the camera thread; the UI runs on the JS thread. The three pieces of state they share are Synchronizables rather than refs, because that is all a worklet can capture:

// src/features/detection/screens/DetectorScreen.tsx
const lastDetect = useMemo(() => createSynchronizable<number>(0), []);
const lastCost = useMemo(() => createSynchronizable<number>(0), []);
const pending = useMemo(() => createSynchronizable<number>(0), []);

The frame processor itself is two gates, one inference and one draw — and the order of the last two is not negotiable:

// Detect BEFORE rendering, never after: render() consumes the frame texture,
// so a resize afterwards hands the model an empty buffer and it returns,
// quite correctly, nothing at all.
const rest = Math.max(DETECT_FLOOR_MS, lastCost.getDirty());
const detect =
  model != null &&
  resizer != null &&
  pending.getDirty() === 0 &&
  Date.now() - lastDetect.getDirty() >= rest;

if (detect) {
  pending.setBlocking(1);
  const started = Date.now();
  const found = readFrameDetections(model!, resizer!, frame);
  const finished = Date.now();
  lastCost.setBlocking(finished - started);
  lastDetect.setBlocking(finished);
  scheduleOnRN(onDetected, found, frame.width, frame.height);
}

render(({ frameTexture, canvas }) => {
  canvas.drawImage(frameTexture, 0, 0);
});

The worklet deliberately does no filtering and no tracking. Both need the frame's dimensions to reach one coordinate system, and both have to stay testable without a camera — so they live on the JS side as plain, unit-tested functions. Model coordinates live in the square the resizer produced, which is not the frame's shape, so getting a box on screen is two conversions: toFrameBox (model square → frame space) then boxToScreen (frame space → screen pixels, undoing the canvas's fit="cover").

Tapping a bracket replaces the camera screen rather than covering it: DetectorScreen returns FaceScanScreen instead of its viewfinder, so the preview, the brackets and the count all unmount and the detector stops. The frame is therefore captured in the tap handler and handed over with ownership — by the time the result screen renders there is no camera left to ask, and it disposes the image when it closes.

Enrolment takes the other route into the same pipeline: it works from a still, so there is no Frame and no resizer, and scanImage.ts draws the snapshot into an offscreen Skia surface and reads the pixels back. That hand-built placement has to agree exactly with what toFrameBox assumes, which is why the two live in the same file and are tested against each other.

Then the request, which the phone can still refuse:

takeSnapshot ─► tight crop (224) ─┐
             ─► mesh ─► pose gate ─┤─► POST /v1/face/verify ─► success + reason
             ─► wide crop (2.7x)  ─┘

A rejection is an ordinary 200 with success: false and a reason (SPOOF, LOW_QUALITY, FACE_NOT_RECOGNIZED); only a malformed request is 4xx. similarity comes back for logging — never to offer a nearest match, which is how a stranger ends up wearing a colleague's name.

Full diagrams in ml-pipeline.md and data-flow.md.

Project structure

Tally/
  index.js               # React Native entry point
  src/
    app/                 # App root: the session + enrolment gate, providers
    features/            # One folder per capability; each owns its screens,
                         #   components, hooks, services and domain types, and
                         #   exposes exactly one index.ts to the others
      detection/         #   YOLO26, the frame processor, NMS, tracking, box
                         #   geometry, camera controls, DetectorScreen
      recognition/       #   FaceMesh, the pose gate, crop geometry, the face
                         #   service client, enrolment and the scan preview
      history/           #   Scan records, MMKV store, cloud repository, retry
                         #   queue, thumbnails, HistorySheet
      auth/              #   Supabase session, sign-in, registration
      settings/          #   Preferences and the screen that changes them
      export/            #   History to CSV
    core/                # Infrastructure that has no idea what a face is
      ml/                #   Pixels into a tensor, in a given layout and range
      storage/           #   The one shared MMKV instance
      supabase/          #   The Supabase client
    shared/              # Generic UI, hooks, i18n, theme, geometry types
  assets/
    fonts/               # Geist (SIL OFL), linked with react-native-asset
    models/              # widerfaceyolo26.tflite, FaceMesh, and notes on their
                         #   verified tensor layouts
  docs/                  # Architecture overview, dependency rules, 6 ADRs
  tools/                 # inspect_tflite.py — reads shapes out of a FlatBuffer
  __mocks__/             # Jest replacements for native modules
  patches/               # patch-package patch for react-native-fast-tflite
  fastlane/              # Build and release lanes
  android/               # Android project (bare workflow)
  ios/                   # iOS project + Podfile

Tests live beside the code they cover, in __tests__/ folders inside each feature.

Dependency rules

app      ──►  features, core, shared
features ──►  core, shared, other features' PUBLIC API
core     ──►  shared
shared   ──►  core

Enforced in .eslintrc.js with ESLint's built-in no-restricted-imports — no plugin, no extra dependency — and npm run verify starts with eslint ., so a violation fails CI. Inside a feature imports are relative; across features they use the @features/… alias, which is what makes the rule a single pattern rather than six exception lists.

import { trackFaces } from '../tracking/tracker';                    // fine
import { EnrolFaceScreen } from '@features/recognition';             // fine — public API
import { readMesh } from '@features/recognition/pipeline/faceMesh';  // ERROR

Path aliases (@app, @features, @core, @shared, @) are declared three times — babel.config.js, tsconfig.json and jest.config.js — because none of the three reads the others. Metro takes its resolution from Babel, so it needs no fourth copy.

Testing

npm test

14 suites, co-located with the code. They cover the places where a mistake produces no error at all, just quietly wrong results:

• parseDetections.test.js — the raw YOLO head decode, above all the channel-major indexing (c * 2100 + a, not a * 5 + c): transpose those and boxes still appear, just in the wrong places
• letterbox.test.js / boxLayout.test.js — both coordinate conversions, including modelDestRect() agreeing with toFrameBox() across aspect ratios, which is what keeps the still-image path aligned with the camera path
• detections.test.js — thresholds and NMS merging
• tracker.test.js — id stability across rounds, surviving a missed round, and the identity guarantee that stops the overlay re-rendering when nothing moved
• faceCrops.test.js / meshLandmarks.test.js — crop geometry (including the 2.7× context window clamped inside the frame) and the landmark maths, kept free of Skia so they can be tested at all
• pose.test.js — the gate that decides a face never leaves the device
• historyRecord.test.js — day grouping and weekTotals(), above all the window edges. Both count back with setDate rather than subtracting 86400000, because a day is not always 24 hours — across a US spring-forward, subtracting six days lands at 23:00 on the previous day
• syncQueue.test.js — the offline retry queue, including the invariant that keeps a deleted scan deleted: queueing a delete drops that id from the upload queue, so a scan removed before its upload landed is not recreated in the cloud by the next flush
• exportHistory.test.js — CSV export
• i18n.test.js / i18nRuntime.test.js — the half of the translation contract tsc cannot see (placeholders, plural forms, the two catalogs agreeing key for key), and the runtime switching machinery
• dialog.test.js — a component test that doubles as a canary for the Reanimated Jest setup

Native modules under Jest

Two pieces of configuration in jest.config.js exist only because of native packages:

• transformIgnorePatterns — reanimated, worklets, skia, vision-camera, nitro, blur and camera-roll all ship ESM. Note the pattern also accepts \ for Windows paths, and must not put / after react-native or it misses react-native-reanimated.
• resolver: 'react-native-worklets/jest/resolver.js' — worklets calls into its native module as an import-time side effect, and under Jest there is no binary to answer. The bundled resolver drops the .native extensions so the call is never reached, which is preferable to mocking reanimated wholesale: the real library loads fine once worklets stops reaching for the binary, so animated components can be rendered and asserted on rather than stubbed out.

MMKV needs no mapping — createMMKV() detects JEST_WORKER_ID itself and returns an in-memory store.

Permissions

Permission	Platform	Why it's needed
CAMERA / NSCameraUsageDescription	Android, iOS	Frame source for scanning
VIBRATE	Android	Haptic alert when a scan records itself
INTERNET	Android	Metro in debug builds; Supabase auth, history sync and recognition in every build
NSPhotoLibraryUsageDescription / NSPhotoLibraryAddUsageDescription	iOS	Saving a scan's image to the photo library

Release builds

Release enables R8 minification. proguard-rules.pro keeps org.tensorflow.lite.** and com.google.ai.edge.litert.** — those classes are reached only through JNI FindClass, so R8 cannot see the references and would otherwise strip them, crashing the app on model load.

Signing reads four Gradle project properties, supplied as ORG_GRADLE_PROJECT_* environment variables — Gradle's own convention for passing a project property through the environment, and one that keeps passwords out of the process list unlike -P flags:

Variable	Value
ORG_GRADLE_PROJECT_TALLY_STORE_FILE	path to the keystore
ORG_GRADLE_PROJECT_TALLY_STORE_PASSWORD	keystore password
ORG_GRADLE_PROJECT_TALLY_KEY_ALIAS	key alias (not a secret)
ORG_GRADLE_PROJECT_TALLY_KEY_PASSWORD	key password

fastlane loads them from fastlane/.env (gitignored) and the Gradle subprocess inherits them; CI sets the same names from repository secrets, so a local release and a CI release are signed through one mechanism instead of two. Keep the keystore itself outside the repo — *.keystore and *.jks are gitignored.

Running ./gradlew directly bypasses the .env loading; put the same four properties in ~/.gradle/gradle.properties if you want that path to sign too. Without any of them the release build falls back to the debug key, so it still builds and installs for local testing — it just cannot be published.

Fastlane

The build commands live in fastlane/Fastfile so a local release and a CI release run the same code path. Requires Ruby:

bundle install

Lane	npm script	What it does
fastlane android install	—	debug APK, installed on the connected device
fastlane android release	npm run release	signed APKs, one per phone ABI
fastlane android bundle	npm run release:aab	App Bundle for the Play Store
fastlane android github tag:v1.0.0	npm run release:github	the release lane, then publish to GitHub

Prefix the lanes with bundle exec. The install lane has no npm script because npm run android already builds, installs, and starts Metro.

What the lanes run underneath

assembleRelease on its own produces one universal APK carrying all four ABIs — around 211MB, of which ~98MB is the x86/x86_64 libraries only an emulator ever loads. -PsplitApks emits one APK per phone ABI instead:

cd android && ./gradlew clean

cd android && ./gradlew assembleRelease -PsplitApks

Two invocations, not ./gradlew clean assembleRelease. Combined, clean deletes the autolinked libraries' prefab_package directories while the task graph already assumes they exist, and the CMake configure step dies on prefab: directory … is not readable. The release lane runs them separately for this reason.

The clean itself matters: React Native's asset-copy task only ever adds to android/app/build/generated/res/react/, so a model removed from assets/models keeps shipping in the APK until that directory is wiped.

Measured on a clean build: 73MB for arm64-v8a, 59MB for armeabi-v7a, down from a 211MB universal APK.

Play splits an AAB per-ABI on the server, so bundleRelease needs no ABI configuration and no per-APK versionCode juggling:

cd android && ./gradlew bundleRelease

Publishing

Pushing a v* tag runs release.yml, which calls the github lane to build both APKs and attach them to a GitHub Release. It needs five repository secrets, and fails fast if any is missing rather than publishing a debug-signed or backend-less build:

Secret	Value
TALLY_KEYSTORE_BASE64	base64 -w0 tally-release.jks
TALLY_STORE_PASSWORD	keystore password
TALLY_KEY_PASSWORD	key password
TALLY_SUPABASE_URL	same value as SUPABASE_URL in your local .env
TALLY_SUPABASE_ANON_KEY	same value as SUPABASE_ANON_KEY in your local .env

The workflow writes the last two into a .env file before the build, since react-native-dotenv inlines them into the bundle at bundle time — without them the release APK would build fine and only fail at sign-in. The keystore is shredded from the runner afterwards, if: always().

Generate the keystore once and keep it safe — losing it means losing the ability to ship updates to anyone who already installed the app:

keytool -genkeypair -v -keystore tally-release.jks -alias tally -keyalg RSA -keysize 2048 -validity 10000

The APK itself never belongs in the repository: GitHub caps files at 100MB, and a binary committed once stays in the history for every future clone.

Engineering notes

• Three constants must agree with a binary file, and nothing checks that they do. MODEL_SIZE, the resizer's pixelLayout, and the expected output rank. TFLite accepts a wrongly sized buffer without a word, reads part of it, and returns numbers that are not detections — a face filling the frame reads as zero, with no error anywhere. This has already happened once: the model file was swapped for a 320 export while the constant still said 640, and the resizer went on producing 1,228,800 floats for a model that wanted 307,200. If detections stop entirely, check the shape before anything else.

  python tools/inspect_tflite.py assets/models/widerfaceyolo26.tflite

• android:largeHeap="true" is required, not decorative. Two TFLite models plus Skia. react-native-fast-tflite reads each model file whole into a Java byte array before handing it to native, so loading a second copy of the same model in another screen was enough to OOM. Both models are loaded once, in DetectorScreen, and passed down.

• Pixel layout depends on how the model was exported, not on which model it is. The detector is NCHW and FaceMesh is NHWC. Feed a model the wrong layout and it still runs and still returns numbers, just meaningless ones.

• The GPU delegate is OFF, and that is a measurement. It works — 411/411 nodes delegated — and it runs the model 1.7× faster. It is still the wrong choice here, because Skia draws the preview on the same GPU: inference and rendering fight, and the render loses in visible chunks. Same build, same scene, only TRY_GPU changed: 90th-percentile frame time 69 ms on the GPU vs 22 ms on the CPU, and the GPU histogram is bimodal with a second cluster at 53–97 ms sitting exactly on the cost of a pass. That cluster is a visible periodic jolt; on the CPU it does not exist. The cost is detection dropping from 3.2 to 2.5 passes/second. Worth re-measuring on hardware whose GPU is not also the rendering bottleneck.

• Zoom may only be set after onStarted. Setting it earlier makes CameraX throw Camera is not active; the OperationCanceledException raised while the camera session restarts is harmless and swallowed deliberately.

• Never write a Reanimated shared value in a render body. Strict mode warns about it, and the fix is always an effect keyed on the prop that drives the animation. Reading .value during render counts too.

• Animate transforms, not layout props. The detection brackets glide with translateX/Y on a memoised component, so a moving face costs the compositor a matrix rather than costing React a layout pass on every detection round.

• The worklet applies only a hard floor (RAW_SCORE_FLOOR), not the configured threshold. Everything above the floor is shipped to JS and filtered there, so the floor must stay below any DETECT_THRESHOLD worth setting. The threshold is deployment config, not a preference: it is tuned once against this model and the rooms the app is used in, and a user who can move it can only make their own counts wrong.

• A Modal is its own window on Android, with consequences in three places. A Skia <Canvas> inside one draws nothing, which is why modalIcons.tsx and Checkbox.tsx redraw their glyphs from plain Views; stacking a second Modal means two windows fighting over the back button, which is why both Dialog and the photo viewer are absolutely positioned overlays; and the window on top owns the back press, so Dialog's own BackHandler only fires on a plain screen.

• The Skia label font must be a family that really exists on the device. 'System', 'Roboto' and the empty string all return a Typeface that looks valid but has no glyphs — text measures to width = 0 and draws invisibly. On Android only 'sans-serif' works.

Key libraries

Library	Role
react-native-vision-camera	Camera, permissions, zoom, focus
react-native-vision-camera-skia	SkiaCamera — frame rendering through Skia, takeSnapshot()
react-native-vision-camera-resizer	GPU-accelerated frame resize to the model's input size
react-native-fast-tflite	Loading and running .tflite via runSync inside the worklet
react-native-worklets	createSynchronizable, scheduleOnRN — the JS ↔ worklet bridge
@shopify/react-native-skia	Drawing the overlay, offscreen surfaces, encoding crops
react-native-nitro-image	Writing Skia image bytes out to a temporary file
react-native-reanimated	HUD and bracket animations
@react-native-community/blur	Frosted-glass backgrounds for the HUD cards
@supabase/supabase-js	Auth and the Postgres/storage backend for history sync
axios	The face service client: base URL, JWT interceptor, timeout, and every failure flattened into one error type
react-native-mmkv	The one on-device key/value store — history, settings, locale, and the Supabase session
i18n-js	Translation lookup, interpolation and locale fallback

Documentation

Document	What it covers
overview.md	The four layers and the six features
dependency-rules.md	What may import what, and how it is enforced
feature-boundaries.md	Who owns what, and the calls that were not obvious
ml-pipeline.md	Threading, tensor shapes, measured performance
data-flow.md	Counting, naming, restoring, and offline behaviour
adr/	Six decision records, with the reasoning
assets/models/README.md	Verified tensor layouts for both bundled models

License

AGPL-3.0.

The licence is not a free choice here: the bundled Ultralytics YOLO26 weights are themselves AGPL-3.0, and that obligation reaches the whole app. In practice that means anyone you distribute a build to — including over a network — is entitled to the corresponding source. If you need to ship a closed-source build, the route is a commercial licence from Ultralytics, not a different licence on this repository.

Geist, in assets/fonts, is licensed separately under the SIL Open Font License.