diff --git a/README.md b/README.md index c601dff..79c9333 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ Cookie Voting is a full-stack application designed to facilitate cookie competit - **Admin Dashboard**: Create and manage voting events, upload images, and tag cookies - **Responsive Design**: Works seamlessly on desktop and mobile devices - **Component Documentation**: Comprehensive Storybook documentation for all UI components + > A modern web application for managing and voting on cookie competitions. Powered by AI cookie detection, Firebase, and React. @@ -27,8 +28,6 @@ Cookie Voting is a full-stack application designed to facilitate cookie competit - Node.js 20+ and npm - Firebase CLI (for deployment) - Google Cloud Vision API enabled (for cookie detection) -- **Node.js 20+** -- **Firebase CLI** ### Installation @@ -54,7 +53,7 @@ npm run dev ## ✨ Key Features -- 🤖 **AI-Powered Detection**: Automatically detects individual cookies in tray images using Google Gemini AI. +- 🤖 **AI-Powered Detection**: Automatically detects individual cookies in tray images using **Google Cloud Vision API**. - 🗳️ **Interactive Voting**: Real-time voting system with multiple categories. - 📊 **Live Results**: Instant tallying and leaderboard updates. - 🛡️ **Admin Dashboard**: Comprehensive tools for event management and image tagging. @@ -67,30 +66,23 @@ npm run dev Detailed documentation is available for developers and contributors: - **[Product Requirements (PRD)](./docs/PRD.md)**: Feature specs and user stories. -- **[Architecture & Tech Stack](#tech-stack)**: Overview of the system. -- **[Gemini AI Setup](./docs/GEMINI_SETUP.md)**: Configuring the vision API. +- **[Technical Architecture](./docs/ARCHITECTURE_TECHNICAL.md)**: Deep dive into the stack and core workflows. +- **[Vision API Pipeline](./docs/VisionAPI.md)**: Details on the AI detection and cropping system. - **[Emulator Setup](./docs/EMULATOR_SETUP.md)**: Running Firebase locally. - **[Testing Guide](./docs/TESTING_GUIDE.md)**: Strategy for Unit, E2E, and Visual tests. - **[Storybook Library](https://tytaniumdev.github.io/CookieVoting/)**: Component documentation. - **[Deployment](./.github/DEPLOYMENT_SETUP.md)**: CI/CD configuration. +- **[Gemini Script Setup](./docs/GEMINI_SCRIPT_SETUP.md)**: (Experimental) Setup for the Gemini utility script. -- **[Storybook Component Library](https://tytaniumdev.github.io/CookieVoting/)** - Browse and interact with all UI components -- [Product Requirements Document (PRD)](./docs/PRD.md) - Source of truth for functionality -- [GEMINI_SCRIPT_SETUP.md](./docs/GEMINI_SCRIPT_SETUP.md) - Setup guide for experimental Gemini AI cookie detection script -- [EMULATOR_SETUP.md](./docs/EMULATOR_SETUP.md) - Local Firebase emulator setup -- [STORYBOOK_SETUP.md](./docs/STORYBOOK_SETUP.md) - Storybook development guide -- [.github/DEPLOYMENT_SETUP.md](./.github/DEPLOYMENT_SETUP.md) - Deployment configuration ## 🛠️ Tech Stack - **Frontend**: React 19, TypeScript, Vite, TailwindCSS - **Backend**: Firebase (Auth, Firestore, Storage, Functions) -- **AI Detection**: Google Cloud Vision API +- **AI Detection**: Google Cloud Vision API (Production), Google Gemini (Experimental Script) - **UI Components**: Custom components with Storybook documentation - **Testing**: Vitest, Playwright, Storybook -- **AI**: Google Gemini API -- **Testing**: Vitest, Playwright - **CI/CD**: GitHub Actions ## 🤝 Contributing diff --git a/docs/ARCHITECTURE_TECHNICAL.md b/docs/ARCHITECTURE_TECHNICAL.md new file mode 100644 index 0000000..7e49966 --- /dev/null +++ b/docs/ARCHITECTURE_TECHNICAL.md @@ -0,0 +1,113 @@ +# Technical Architecture & Deep Dive + +> **Audience:** Developers, Architects, Contributors. +> **Scope:** Full-stack overview of the Cookie Voting application. + +## 1. System Overview + +Cookie Voting is a serverless application built on **Google Firebase**. It leverages **React 19** for the frontend and **Cloud Functions for Firebase** for backend logic. The core innovation is the **Assisted Cookie Detection Pipeline**, which uses Google Cloud Vision API to automate the identification of cookies in tray images. + +### Tech Stack + +- **Frontend**: React 19, TypeScript, Vite, TailwindCSS, Zustand (State Management). +- **Backend**: Firebase Cloud Functions (Node.js 20). +- **Database**: Cloud Firestore (NoSQL). +- **Storage**: Cloud Storage for Firebase. +- **AI/ML**: Google Cloud Vision API (Production), Google Gemini (Experimental/Script only). +- **Auth**: Firebase Authentication (Anonymous for voters, Email/Password for admins). + +--- + +## 2. Core Workflows + +### 2.1 Assisted Cookie Detection Pipeline (The "Crop" Flow) + +This is the most complex flow in the system, involving multiple cloud services. + +1. **Upload**: Admin uploads a tray image (containing multiple cookies) to the "Cookie Cropper" UI. +2. **Trigger**: Image is saved to `uploads/{batchId}/original.jpg` in Storage. +3. **Detection (Async)**: + - `processCookieImage` Cloud Function is triggered. + - It sends the image to **Google Cloud Vision API** (Object Localization). + - Detected bounding boxes are saved to the `cookie_batches/{batchId}` document in Firestore under `detectedObjects`. + - Batch status updates to `review_required`. +4. **Review**: Admin sees the detected boxes on the original image in the UI. They can resize, add, or remove boxes. +5. **Confirmation**: + - Admin clicks "Confirm Crops". + - UI calls the `confirmCookieCrops` Callable Function with the final coordinates. +6. **Processing**: + - The function uses `sharp` to physically crop the original image into individual cookie images. + - Individual images are uploaded to `processed_cookies/{batchId}/{cookieId}.jpg`. + - The `Category` document in Firestore is updated with the new cookie list. + - Batch status updates to `ready`. + +### 2.2 Voting Flow + +1. **Join**: User visits the event URL. An anonymous Firebase Auth session is established. +2. **Vote**: User selects their favorite cookies. Votes are stored locally in the `UserVote` store. +3. **Submit**: User submits votes. A `UserVote` document is created in `events/{eventId}/votes/{userId}`. +4. **Results**: Real-time listeners on the `votes` collection aggregate results (client-side for small scale, or via potential future aggregation functions). + +--- + +## 3. Data Model (Firestore) + +### `events/{eventId}` +Stores the configuration for a voting event. +- `name`: String +- `status`: 'voting' | 'completed' | 'draft' +- `adminCode`: String (for simplified admin access, if used) + +### `events/{eventId}/categories/{categoryId}` +Represents a "Tray" or category of cookies (e.g., "Most Creative"). +- `name`: String +- `cookies`: Array of Objects + - `id`: String (unique cookie ID) + - `imageUrl`: String (public URL) + - `baker`: String (optional) + +### `events/{eventId}/votes/{userId}` +Stores a single user's votes to ensure one-vote-per-person per event. +- `votes`: Map + +### `cookie_batches/{batchId}` +Temporary state for the image processing pipeline. +- `status`: 'uploading' | 'processing' | 'review_required' | 'ready' | 'error' +- `originalImageRef`: String (Storage path) +- `detectedObjects`: Array of Objects (Vision API results) + - `normalizedVertices`: Array of {x, y} (0-1 coordinates) + - `confidence`: Number + +--- + +## 4. Cloud Functions + +Located in `functions/src/index.ts`. + +| Function Name | Type | Purpose | +| :--- | :--- | :--- | +| `processCookieImage` | **Trigger** (Storage) | Listens for new uploads, calls Vision API, initializes batch data. | +| `confirmCookieCrops` | **Callable** (HTTPS) | Takes confirmed coordinates, crops images using `sharp`, updates the database. | +| `addAdminRole` | **Callable** (HTTPS) | Grants `admin: true` custom claim to a user (Admin only). | +| `removeAdminRole` | **Callable** (HTTPS) | Removes admin privileges. | + +--- + +## 5. Security & Access Control + +### Custom Claims +The system uses Firebase Custom Claims to designate admins. +- `token.admin === true`: Grants full read/write access to all collections and ability to invoke admin functions. + +### Firestore Rules +- **Public Read**: Most data (`events`, `categories`, `cookies`) is readable by anyone (allow `read: if true`). +- **Admin Write**: strict `allow write: if isGlobalAdmin()` for core content. +- **Vote Write**: Open write access for `votes` collection (rate limiting/validation handled via app logic and potential future security rules enhancements). +- **Batch Isolation**: `cookie_batches` is largely managed by backend functions, with limited admin write access for status updates. + +--- + +## 6. Experimental Features + +### Gemini Script (`scripts/detect-all-images.js`) +An experimental utility script exists to batch-process images using **Google Gemini AI**. This is **NOT** part of the production pipeline and is intended for local testing or bulk backfilling of data in development environments. See `docs/GEMINI_SCRIPT_SETUP.md` for details. diff --git a/docs/VisionAPI.md b/docs/VisionAPI.md index 1227d1c..bb85657 100644 --- a/docs/VisionAPI.md +++ b/docs/VisionAPI.md @@ -1,359 +1,83 @@ -Production Plan: Automated Cookie Segmentation Pipeline - -Architecture: Firebase Cloud Functions + Google Cloud Vision API - -1. Executive Summary - -This document outlines the implementation plan for an automated "Upload-to-Crop" pipeline. The goal is to allow users to upload a single group photo of Christmas cookies, automatically detect individual cookies using AI, crop them into high-resolution isolated images, and prepare them for a voting interface. - -Key Technology: - -Firebase Storage: Image hosting. - -Cloud Functions (Node.js): Backend logic triggers. - -Google Cloud Vision API: Object localization (AI). - -Sharp (npm): High-performance image processing/cropping. - -Firestore: Database for voting and metadata. - -2. Architecture & Data Flow - -User Action: User uploads batch_01/raw_tray.jpg to Firebase Storage. - -Trigger: functions.storage.object().onFinalize fires. - -Analysis: Function sends image to Vision API. - -Response: Vision API returns bounding box coordinates for objects (cookies). - -Processing: Function downloads the image and uses Sharp to slice it into cookie_01.jpg, cookie_02.jpg, etc. - -Storage: Cropped images are uploaded to processed_cookies/batch_01/. - -Database: Metadata (URLs, initial vote counts) is written to Firestore. - -UI Update: Frontend listens to Firestore and displays the new cookies in real-time. - -3. Prerequisites & Configuration - -A. Google Cloud Console - -Project Plan: Ensure your Firebase project is on the Blaze (Pay as you go) plan. - -Why? Node.js Cloud Functions cannot make external network requests (even to Google APIs) on the Spark plan. - -Cost Note: You will likely stay within the free tier limits (see Section 7), but the plan upgrade is technically required to unlock the capability. - -Enable API: - -Go to Google Cloud Console. - -Select your project. - -Navigate to APIs & Services > Library. - -Search for "Cloud Vision API" and click Enable. - -B. Environment Setup - -Ensure you have the Firebase CLI installed and initialized: - -npm install -g firebase-tools -firebase login -firebase init functions -# Select JavaScript (or TypeScript if preferred, this guide uses JS) - - -4. Data Modeling - -Storage Structure - -uploads/{batchId}/original.jpg (Input) - -processed_cookies/{batchId}/{cookieId}.jpg (Output) - -Firestore Schema - -Collection: cookie_batches - -Document: {batchId} - -status: "processing" | "ready" | "error" - -createdAt: Timestamp - -originalImageRef: String - -Subcollection: candidates - -Document: {cookieId} - -storagePath: "processed_cookies/..." - -detectedLabel: "Cookie" (or "Food", "Baked Goods") - -confidence: Number (0.0 - 1.0) - -votes: 0 - -5. Implementation Details - -Step 1: Dependencies - -Navigate to your functions folder and install the required packages. - -cd functions -npm install firebase-admin firebase-functions @google-cloud/vision sharp fs-extra - - -Step 2: The Cloud Function Code (functions/index.js) - -const functions = require("firebase-functions"); -const admin = require("firebase-admin"); -const vision = require("@google-cloud/vision"); -const sharp = require("sharp"); -const path = require("path"); -const os = require("os"); -const fs = require("fs-extra"); - -admin.initializeApp(); -const db = admin.firestore(); -const storage = admin.storage(); - -// Initialize Vision Client -const visionClient = new vision.ImageAnnotatorClient(); - -// CONFIGURATION -const TARGET_COLLECTION = "cookie_batches"; -// Padding adds "breathing room" around the cookie so it isn't cropped too tightly. -// 0.1 = 10% extra width/height added to the crop. -const PADDING_PERCENTAGE = 0.1; - -exports.processCookieImage = functions.storage.object().onFinalize(async (object) => { - const fileBucket = object.bucket; - const filePath = object.name; - const contentType = object.contentType; - - // --- 1. Validation --- - if (!contentType.startsWith("image/")) return console.log("Not an image."); - if (!filePath.startsWith("uploads/")) return console.log("Not in uploads folder."); - if (filePath.includes("processed_cookies")) return console.log("Already processed."); - - const fileName = path.basename(filePath); - // Assumes path format: uploads/{batchId}/original.jpg - const batchId = path.dirname(filePath).split(path.sep).pop(); - - const bucket = storage.bucket(fileBucket); - const tempFilePath = path.join(os.tmpdir(), fileName); - - // Initialize Batch Status in Firestore - const batchRef = db.collection(TARGET_COLLECTION).doc(batchId); - await batchRef.set({ - status: "processing", - createdAt: admin.firestore.FieldValue.serverTimestamp(), - originalImage: filePath - }, { merge: true }); - - try { - // --- 2. Download --- - await bucket.file(filePath).download({ destination: tempFilePath }); - - // --- 3. Vision API (Object Localization) --- - const [result] = await visionClient.objectLocalization(tempFilePath); - const objects = result.localizedObjectAnnotations; - - if (!objects || objects.length === 0) { - console.log("No objects detected."); - await batchRef.update({ status: "empty" }); - return; - } - - // Get image dimensions for pixel math - const metadata = await sharp(tempFilePath).metadata(); - const imgWidth = metadata.width; - const imgHeight = metadata.height; - - const uploadPromises = []; - let cookieCounter = 0; - - // --- 4. Processing Loop --- - for (const object of objects) { - // Optional: Filter non-food items here if needed - // if (!['Food', 'Baked goods', 'Cookie', 'Snack'].includes(object.name)) continue; - - // Calculate Bounding Box - const vertices = object.boundingPoly.normalizedVertices; - - // Find min/max (0-1 range) - let minX = 1, maxX = 0, minY = 1, maxY = 0; - vertices.forEach(v => { - minX = Math.min(minX, v.x); - maxX = Math.max(maxX, v.x); - minY = Math.min(minY, v.y); - maxY = Math.max(maxY, v.y); - }); - - // Convert to Pixels & Apply Padding - let left = Math.floor(minX * imgWidth); - let top = Math.floor(minY * imgHeight); - let width = Math.floor((maxX - minX) * imgWidth); - let height = Math.floor((maxY - minY) * imgHeight); - - const padX = Math.floor(width * PADDING_PERCENTAGE); - const padY = Math.floor(height * PADDING_PERCENTAGE); - - // Ensure we don't crop outside the image - left = Math.max(0, left - padX); - top = Math.max(0, top - padY); - width = Math.min(imgWidth - left, width + (padX * 2)); - height = Math.min(imgHeight - top, height + (padY * 2)); - - // --- 5. Crop Image --- - const outputFileName = `cookie_${Date.now()}_${cookieCounter++}.jpg`; - const outputTempPath = path.join(os.tmpdir(), outputFileName); - - await sharp(tempFilePath) - .extract({ left, top, width, height }) - .toFile(outputTempPath); - - // --- 6. Upload Crop --- - const destination = `processed_cookies/${batchId}/${outputFileName}`; - - const uploadTask = bucket.upload(outputTempPath, { - destination: destination, - metadata: { - contentType: 'image/jpeg', - metadata: { parentBatch: batchId, label: object.name } - } - }).then(async (uploadResult) => { - // --- 7. Save to Firestore --- - await batchRef.collection("candidates").add({ - storagePath: destination, - detectedLabel: object.name, - confidence: object.score, - votes: 0, - createdAt: admin.firestore.FieldValue.serverTimestamp() - }); - return fs.remove(outputTempPath); // Cleanup temp crop - }); - - uploadPromises.push(uploadTask); - } - - await Promise.all(uploadPromises); - await batchRef.update({ status: "ready", totalCandidates: cookieCounter }); - - } catch (err) { - console.error(err); - await batchRef.update({ status: "error", error: err.message }); - } finally { - await fs.remove(tempFilePath); // Cleanup original temp - } -}); - - -6. Frontend Integration Guide - -The frontend does not need to know about the API or the cropping logic. It simply uploads a file and listens for the database to change. - -The "Upload & Listen" Pattern - -import { getStorage, ref, uploadBytes } from "firebase/storage"; -import { getFirestore, doc, collection, onSnapshot, setDoc } from "firebase/firestore"; - -// 1. Upload Function -async function uploadTray(file) { - const batchId = Date.now().toString(); // Or a UUID - - // Create the batch doc first so UI can transition to loading state - await setDoc(doc(db, "cookie_batches", batchId), { status: "uploading" }); - - const storageRef = ref(storage, `uploads/${batchId}/original.jpg`); - await uploadBytes(storageRef, file); - - return batchId; -} - -// 2. Listening Function (React Hook Example) -function useCookieCandidates(batchId) { - const [cookies, setCookies] = useState([]); - const [status, setStatus] = useState("loading"); - - useEffect(() => { - if(!batchId) return; - - // Listen to the batch status - const batchUnsub = onSnapshot(doc(db, "cookie_batches", batchId), (doc) => { - setStatus(doc.data()?.status); - }); - - // Listen to the candidates subcollection - const candidatesUnsub = onSnapshot(collection(db, "cookie_batches", batchId, "candidates"), (snapshot) => { - const data = snapshot.docs.map(d => ({id: d.id, ...d.data()})); - setCookies(data); - }); - - return () => { batchUnsub(); candidatesUnsub(); }; - }, [batchId]); - - return { cookies, status }; +# Vision API & Cookie Pipeline + +> **Role:** Technical Implementation Details +> **Context:** See [`ARCHITECTURE_TECHNICAL.md`](./ARCHITECTURE_TECHNICAL.md) for the full system overview. + +This document details the **Assisted Cookie Detection Pipeline**, specifically focusing on the interaction between Firebase Cloud Functions and the Google Cloud Vision API. + +## 1. Overview + +The pipeline automates the difficult task of isolating individual cookies from a group photo ("Tray"). Instead of manual cropping, we use AI to detect objects and suggest crops, which an admin then reviews and confirms. + +**Key Technology:** +- **Google Cloud Vision API**: Service: `objectLocalization`. Returns bounding polygons for detected objects. +- **Sharp (Node.js)**: High-performance image processing library used for contrast enhancement (pre-detection) and final cropping. + +## 2. Implementation: The Two-Step Process + +Unlike a simple "Magic Resize" tool, this pipeline is asynchronous and stateful. + +### Step 1: Detection (The Trigger) +**Function:** `processCookieImage` (in `functions/src/index.ts`) +**Trigger:** `storage.object().onFinalize` (Upload to `uploads/{batchId}/original.jpg`) + +**Logic:** +1. **Validation**: Checks if the file is an image and in the correct path. +2. **Preprocessing**: + - Downloads the image to a temp file. + - Uses `sharp` to **normalize** and **enhance contrast** (linear increase). This significantly improves Vision API detection rates on cookie trays. +3. **Vision API Call**: + - Sends the *enhanced* image to `visionClient.objectLocalization()`. + - Receives a list of `localizedObjectAnnotations` (candidates). +4. **State Update**: + - Writes the raw candidates to `cookie_batches/{batchId}` in the `detectedObjects` field. + - Sets `status: 'review_required'`. + - **Note:** No cropping happens here. We trust the AI to *find* objects, but not to *cut* them perfectly without human review. + +### Step 2: Confirmation (The Action) +**Function:** `confirmCookieCrops` (in `functions/src/index.ts`) +**Trigger:** HTTPS Callable (Admin UI) + +**Logic:** +1. **Input**: Receives `batchId` and a list of `crops` (finalized coordinates). +2. **Processing**: + - Downloads the *original* (un-enhanced) image. + - Iterates through the `crops` list. + - Uses `sharp` to extract the specific region (converting normalized 0-1 coords to pixels). + - Uploads the resulting image to `processed_cookies/{batchId}/{cookieId}.jpg`. +3. **Completion**: + - Updates the target `Category` document with the new cookie URLs. + - Updates `cookie_batches/{batchId}` status to `ready`. + +## 3. Data Structure + +### Batch Document (`cookie_batches/{batchId}`) + +The `detectedObjects` array contains the raw output from Vision API, simplified for the frontend: + +```typescript +interface DetectedObject { + // Normalized coordinates (0 to 1) relative to image dimensions + normalizedVertices: [ + { x: 0.1, y: 0.1 }, // Top-Left + { x: 0.2, y: 0.1 }, // Top-Right + // ... + ]; + confidence: number; // 0.0 to 1.0 } +``` +## 4. Why this approach? -7. Cost & Limits Analysis - -You specifically mentioned using the free tier. Here is the breakdown: - -Service - -Free Tier Limit - -Overage Cost - -Impact on this Feature - -Cloud Vision API - -1,000 units / month - -$1.50 per 1,000 units - -Enough for ~1,000 uploaded tray photos per month. - -Cloud Functions - -2,000,000 invocations / month - -$0.40 per million - -Virtually unlimited for your use case. - -Cloud Storage - -5 GB - -$0.026 / GB - -Images are small; cost is negligible. - -Firestore - -50,000 reads / day - -$0.06 per 100k - -Plenty for standard voting traffic. - -Verdict: This architecture will comfortably run for free for development and small-to-medium scale usage. - -8. Deployment Checklist - -Check Billing: Ensure project is on Blaze plan. - -Enable API: Vision API enabled in Cloud Console. +1. **Accuracy vs. Speed**: We prioritize accuracy. Vision API is good, but "Cookie vs. Brownie" is hard. Human review ensures 100% quality. +2. **Cost Efficiency**: We call the API once per tray, not once per cookie. +3. **Preprocessing**: Real-world photos are often poorly lit. Enhancing contrast *before* sending to AI proved critical during testing. -Deploy Functions: firebase deploy --only functions. +## 5. Troubleshooting -Security Rules: Update Storage/Firestore rules to allow users to read processed_cookies and candidates but only write to uploads and votes. \ No newline at end of file +- **"No objects detected"**: + - Check if the image is too dark. + - Ensure `GOOGLE_APPLICATION_CREDENTIALS` (or default identity) has "Cloud Vision API User" role. + - Check logs for "Vision API detected 0 objects". +- **"Crops are misaligned"**: + - The frontend canvas and the backend `sharp` logic must agree on coordinate space (normalized 0-1 is safest).