Skip to content

ops(worker): add bullmq worker to docker-compose and supervise it #437

Description

@AndreLiar

Problem

The BullMQ worker process (`node --import ./instrument.js workers/index.js`) is not in `docker-compose.production.yml` and is not supervised. It runs as a manually started detached process inside the container.

Consequences:

  • Worker dies on every container restart (CD deploy, crash, OOM kill)
  • All background processing stops silently: file indexing, gap analysis, monitoring jobs, weekly digests
  • BullMQ jobs are left in "active" state with orphaned lock keys → require manual intervention to re-queue
  • The platform gives no alert when this happens

Observed in prod: worker died, 2 documents went unindexed, gap analyses stopped — discovered only when user noticed documents stuck at "uploading".

Fix

Add the worker as a supervised service in `docker-compose.production.yml`:

```yaml
worker:
image: ${BACKEND_IMAGE}
command: node --import ./instrument.js workers/index.js
restart: unless-stopped
env_file: .env
depends_on:
- mongo
- redis
```

Also update the CD workflow (`.github/workflows/cd.yml`) to pull and restart the worker service alongside the backend.

Additional hardening

  • BullMQ stalled job detection (`stalledInterval`) should auto-requeue orphaned jobs after worker death — verify `removeOnFail` and `attempts` are set correctly
  • Add a health-check endpoint or monitoring alert when the worker queue depth exceeds a threshold

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions