Problem
The BullMQ worker process (`node --import ./instrument.js workers/index.js`) is not in `docker-compose.production.yml` and is not supervised. It runs as a manually started detached process inside the container.
Consequences:
- Worker dies on every container restart (CD deploy, crash, OOM kill)
- All background processing stops silently: file indexing, gap analysis, monitoring jobs, weekly digests
- BullMQ jobs are left in "active" state with orphaned lock keys → require manual intervention to re-queue
- The platform gives no alert when this happens
Observed in prod: worker died, 2 documents went unindexed, gap analyses stopped — discovered only when user noticed documents stuck at "uploading".
Fix
Add the worker as a supervised service in `docker-compose.production.yml`:
```yaml
worker:
image: ${BACKEND_IMAGE}
command: node --import ./instrument.js workers/index.js
restart: unless-stopped
env_file: .env
depends_on:
- mongo
- redis
```
Also update the CD workflow (`.github/workflows/cd.yml`) to pull and restart the worker service alongside the backend.
Additional hardening
- BullMQ stalled job detection (`stalledInterval`) should auto-requeue orphaned jobs after worker death — verify `removeOnFail` and `attempts` are set correctly
- Add a health-check endpoint or monitoring alert when the worker queue depth exceeds a threshold
Problem
The BullMQ worker process (`node --import ./instrument.js workers/index.js`) is not in `docker-compose.production.yml` and is not supervised. It runs as a manually started detached process inside the container.
Consequences:
Observed in prod: worker died, 2 documents went unindexed, gap analyses stopped — discovered only when user noticed documents stuck at "uploading".
Fix
Add the worker as a supervised service in `docker-compose.production.yml`:
```yaml
worker:
image: ${BACKEND_IMAGE}
command: node --import ./instrument.js workers/index.js
restart: unless-stopped
env_file: .env
depends_on:
- mongo
- redis
```
Also update the CD workflow (`.github/workflows/cd.yml`) to pull and restart the worker service alongside the backend.
Additional hardening