Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/ARCHITECTURE_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Aggregation is fire-and-forget from the sim PATCH response path. If it fails (ti
- `api/app/api/jobs/[id]/simulations/[simId]/route.ts` — body fields destructured without type validation
- Various PATCH endpoints accept `Record<string, unknown>` without sanitization

**Mitigation:** Add zod schemas for request bodies at system boundaries. Internal service-to-service calls (worker → API) still benefit from validation since Pub/Sub redelivery can send malformed payloads.
**Mitigation:** Add zod schemas for request bodies at system boundaries. Internal service-to-service calls (worker → API) still benefit from validation to prevent malformed payloads.

---

Expand Down Expand Up @@ -148,7 +148,7 @@ Users can submit a job with 4 identical decks. While technically valid (useful f

These cross-cutting patterns contributed to past regressions:

1. **No state machine enforcement** (resolved in PR #78) — Invalid state transitions from Pub/Sub redelivery caused COMPLETED simulations to revert to RUNNING.
1. **No state machine enforcement** (resolved in PR #78) — Invalid state transitions caused COMPLETED simulations to revert to RUNNING.

2. **Silent error swallowing** (resolved in PR #78) — `.catch(() => {})` on fire-and-forget operations hid failures that later caused data inconsistency.

Expand Down
7 changes: 3 additions & 4 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,8 @@ When prompted, type `gcloud` to use the legacy flow. The script will install `jq

### What the Worker Does

- Loads runtime config from Secret Manager (API_URL, GCS_BUCKET, PUBSUB_SUBSCRIPTION, WORKER_SECRET)
- Subscribes to Pub/Sub for job messages
- Loads runtime config from Secret Manager (API_URL, GCS_BUCKET, WORKER_SECRET)
- Polls API to claim jobs
- Runs Forge simulations and POSTs results back to the API
- Receives push commands from the API (config overrides, cancellation, job notifications, drain control) via port 9090

Expand All @@ -112,7 +112,7 @@ The `setup-worker.sh` script places a service account key at `worker/sa.json` an
- **SSH/Tailscale access:** The worker accepts inbound push commands from the API on
port 9090 (config updates, cancellation, job notifications). In local mode this is
localhost-only; in GCP mode it uses VPC-internal networking. The worker also makes
outbound connections to Pub/Sub, API, and Secret Manager.
outbound connections to API and Secret Manager.

### Auto-Deploy with Watchtower

Expand Down Expand Up @@ -155,7 +155,6 @@ The worker can be configured via environment variables (in `.env` or Secret Mana
| `GOOGLE_CLOUD_PROJECT` | GCP Project ID (triggers GCP mode if set) | `magic-bracket-simulator` |
| `API_URL` | Base URL of the API | `http://localhost:3000` or Cloud Run URL |
| `WORKER_SECRET` | Shared secret for worker authentication | `some-secret-string` |
| `PUBSUB_SUBSCRIPTION` | Pub/Sub subscription name (GCP mode only) | `job-created-worker` |
| `SIMULATION_IMAGE` | Docker image for the simulation container | `ghcr.io/tytaniumdev/magicbracketsimulator/simulation:latest` |
| `WORKER_NAME` | Display name for the worker (visible in UI) | Hostname |
| `WORKER_ID` | Unique ID for the worker (auto-generated if unset) | `worker-uuid-1234` |
Expand Down
21 changes: 8 additions & 13 deletions docs/MODE_SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,21 @@ This project supports two operational modes: **LOCAL** and **GCP**. This guide e
|---------|------------|----------|
| Database | SQLite | Firestore |
| File Storage | Local filesystem | Cloud Storage (GCS) |
| Job Queue | Polling-based worker | Pub/Sub |
| Job Queue | Polling-based worker | Polling-based worker |
| Analysis | Win rate / game stats | Win rate / game stats |
| Worker | worker/ (polling) | worker/ (Pub/Sub subscriber) |
| Worker | worker/ (polling) | worker/ (polling) |

## Mode Detection

The system automatically detects the mode based on the `GOOGLE_CLOUD_PROJECT` environment variable:
- **Set**: GCP mode (Firestore, Pub/Sub, GCS)
- **Set**: GCP mode (Firestore, HTTP Polling, GCS)
- **Not set**: LOCAL mode (SQLite, filesystem)

At startup, you'll see log messages like:
```
[Job Store] Running in GCP mode
[Job Store] Project: magic-bracket-simulator
[Job Store] Using: Firestore + Cloud Storage + Pub/Sub
[Job Store] Using: Firestore + Cloud Storage + HTTP Polling
```

## Quick Mode Toggle
Expand Down Expand Up @@ -50,8 +50,7 @@ These commands copy or remove `.env.local` override files in `api/` and `fronten
1. GCP Project with:
- Firestore database
- Cloud Storage bucket
- Pub/Sub topic and subscription
2. Service account key with permissions for Firestore, GCS, and Pub/Sub
2. Service account key with permissions for Firestore and GCS
3. Docker installed (for worker)

### Configuration Files
Expand Down Expand Up @@ -86,7 +85,7 @@ WORKER_API_URL="http://<vm-internal-ip>:9090" # VPC-internal IP for API→worke
npm run dev
```

**Terminal 2: Local Worker (processes jobs via Pub/Sub)**
**Terminal 2: Local Worker**
```bash
npm run worker:gcp
```
Expand All @@ -95,7 +94,7 @@ npm run worker:gcp

| Service | Purpose | Where it runs |
|---------|---------|---------------|
| api | API backend (Firestore/Pub/Sub) | Local or Cloud Run |
| api | API backend (Firestore/Polling) | Local or Cloud Run |
| frontend | React UI | Local or Firebase Hosting |
| worker | Node.js orchestrator that spawns simulation containers (Docker) | Docker on your machine |
| simulation | Java + Forge engine container (ephemeral) | Spawns via `docker run` inside worker |
Expand Down Expand Up @@ -167,10 +166,6 @@ The worker image contains the Node.js orchestrator. The simulation image (`magic
- For local testing, use `http://localhost:3000`
- For Cloud Run, use the deployed URL

### Pub/Sub messages not received
- Check `PUBSUB_SUBSCRIPTION` matches the subscription name in GCP
- Verify service account has `pubsub.subscriber` permission

### Firestore permission errors (e.g. `7 PERMISSION_DENIED: User not authorized to perform this action`)
- **Deployed API (Firebase App Hosting / Cloud Run):** The backend runs as a Cloud Run service account. That identity must have **Cloud Datastore User** so it can read/write Firestore (decks, jobs). Grant the role on the project:
- GCP Console → **IAM & Admin** → **IAM** → find the service account used by your App Hosting backend (e.g. `PROJECT_NUMBER-compute@developer.gserviceaccount.com` or the one shown in **Firebase** → **App Hosting** → your backend → **Settings**).
Expand All @@ -184,6 +179,6 @@ The worker image contains the Node.js orchestrator. The simulation image (`magic
- **Local API:** Ensure service account key path is correct and the key’s account has Firestore read/write (e.g. Cloud Datastore User).

### Jobs stuck in QUEUED
- Ensure worker is running and connected to Pub/Sub
- Ensure worker is running and polling API
- Check worker logs for errors
- Verify `WORKER_SECRET` matches between API and worker
7 changes: 2 additions & 5 deletions docs/SECRETS_SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,6 @@ Use this URL when running `npm run populate-worker-secret`. The frontend always
1. In [GCP Console](https://console.cloud.google.com/iam-admin/serviceaccounts), select your project.
2. Create a service account (e.g. `magic-bracket-worker`) or select an existing one used for the worker.
3. Grant roles (e.g.):
- **Pub/Sub Subscriber** (to pull job messages),
- **Storage Object Admin** (or appropriate GCS role for the artifacts bucket),
- **Cloud Datastore User** (or **Firestore** roles if using Firestore).
4. Open the service account → **Keys** → **Add key** → **Create new key** → **JSON**. Download the JSON file.
Expand All @@ -103,8 +102,6 @@ The **Provision Worker** workflow reads secrets from Doppler and syncs them into
| `WORKER_SECRET` | Shared secret between worker and API |
| `API_URL` | Production API URL (see §0) |
| `GCS_BUCKET` | Cloud Storage bucket name (e.g. `magic-bracket-simulator-artifacts`) |
| `PUBSUB_SUBSCRIPTION` | Pub/Sub subscription for jobs (e.g. `job-created-worker`) |
| `PUBSUB_WORKER_REPORT_IN_SUBSCRIPTION` | Pub/Sub subscription for worker report-in (e.g. `worker-report-in-worker`) |
| `GHCR_USER` | GitHub username (for Watchtower to pull worker images) |
| `GHCR_TOKEN` | GitHub PAT with `read:packages` scope (for Watchtower) |

Expand All @@ -131,7 +128,7 @@ Use `--defaults` for non-interactive mode: `npm run populate-worker-secret -- --

#### IAM requirements

The identity the worker uses (ADC or service account key) must have **Secret Manager Secret Accessor** on the project. Same key can have Pub/Sub, GCS, Firestore roles as before. For the provision workflow, the SA key also needs **Secret Manager Admin** (to create/update secrets).
The identity the worker uses (ADC or service account key) must have **Secret Manager Secret Accessor** on the project. Same key can have GCS, Firestore roles as before. For the provision workflow, the SA key also needs **Secret Manager Admin** (to create/update secrets).

---

Expand Down Expand Up @@ -178,7 +175,7 @@ Firebase client config (API key, project ID, etc.) is still set at **build time*

Set at least:

- `GOOGLE_CLOUD_PROJECT`, `GCS_BUCKET`, `PUBSUB_TOPIC`, and any secrets already used.
- `GOOGLE_CLOUD_PROJECT`, `GCS_BUCKET`, and any secrets already used.
- **WORKER_SECRET** (optional but recommended): a shared secret string. Set the same value in:
- Cloud Run (API) env,
- worker config (in Secret Manager via `npm run populate-worker-secret`, or in `.env` if not using Secret Manager).
Expand Down
5 changes: 2 additions & 3 deletions docs/STALE_SWEEPER.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ For each `QUEUED` or `RUNNING` job, the endpoint:
conditional writes so a worker completing a sim at the last millisecond
always wins the race.
3. **Runs the existing recovery path** (`recoverStaleJob`) to re-publish
stale-PENDING Pub/Sub messages and re-trigger aggregation.
stale-PENDING messages and re-trigger aggregation.
4. **Explicitly triggers aggregation** for local-mode jobs where every sim
is now terminal (Firestore mode's recovery already does this).

Expand Down Expand Up @@ -83,5 +83,4 @@ and redeploy.

See `docs/superpowers/specs/2026-04-10-stuck-job-prevention-design.md` and
the incident writeup for job `uxBSYQvYB4JNoycuLSzz`, which sat stuck at
24/25 sims for ~12 hours because of a worker restart plus Pub/Sub
starvation.
24/25 sims for ~12 hours because of a worker restart plus starvation.
Loading