Environment
- OpenShip v0.6.6 (
ghcr.io/oblien/openship-api:0.6.6), self-hosted, Bun runtime
- Database service: MySQL 9.7, ~110GB datadir (mysqldump stream is tens of GB)
- Destination: SFTP (also reproduced with the local destination)
Symptom
A managed backup policy run against a large database fails after ~60–70s with:
RangeError: Out of memory
at ... docker-modem demuxer
Small databases back up fine. The failure is deterministic on large ones — ours dies ~64s in, every run, after the API process balloons in RSS.
Root cause (from reading the path the bytes take)
Every backup byte is routed through the API process: docker exec via dockerode → docker-modem demuxer → destination writer (ssh2 SFTP / local fs).
Two compounding problems:
- docker-modem's stream demuxer does not apply backpressure under Bun. The exec socket is read as fast as Docker can produce (a fast
mysqldump easily does hundreds of MB/s), and demuxed chunks are buffered in JS heap when the destination writer is slower.
- The destination side drains far slower than the source produces. ssh2's SFTP writeStream drains at ~1.7 MB/s in our measurements. At ~100+ MB/s in and ~1.7 MB/s out, the process accumulates ~6 GB+ of buffered chunks per minute →
RangeError: Out of memory at roughly the 64s mark.
So the OOM is not the dump size per se — it's unbounded buffering between a fast reader and a slow writer.
Suggested fixes (any of these)
- Respect backpressure end-to-end: pause the exec stream when the destination writer's
write() returns false, resume on drain (or pipe through a bounded Transform/pipeline() with a small highWaterMark).
- Alternatively, don't route the payload through the API at all: run the dump on the host/server side and hand the destination a file or a bounded pipe (e.g.
docker exec … mysqldump | zstd > … executed via the host channel), keeping the API as control plane only.
- At minimum: document a size ceiling for managed policies and fail fast with a clear error instead of OOMing the API process.
Workaround we use
A custom job (POST /api/jobs) that runs docker exec <db> mysqldump … | zstd directly to the destination storage on the server. Works fine for the same 110GB database, which supports the "control plane vs data plane" fix above.
Environment
ghcr.io/oblien/openship-api:0.6.6), self-hosted, Bun runtimeSymptom
A managed backup policy run against a large database fails after ~60–70s with:
Small databases back up fine. The failure is deterministic on large ones — ours dies ~64s in, every run, after the API process balloons in RSS.
Root cause (from reading the path the bytes take)
Every backup byte is routed through the API process:
docker execvia dockerode → docker-modem demuxer → destination writer (ssh2 SFTP / local fs).Two compounding problems:
mysqldumpeasily does hundreds of MB/s), and demuxed chunks are buffered in JS heap when the destination writer is slower.RangeError: Out of memoryat roughly the 64s mark.So the OOM is not the dump size per se — it's unbounded buffering between a fast reader and a slow writer.
Suggested fixes (any of these)
write()returns false, resume ondrain(or pipe through a boundedTransform/pipeline()with a small highWaterMark).docker exec … mysqldump | zstd > …executed via the host channel), keeping the API as control plane only.Workaround we use
A custom job (
POST /api/jobs) that runsdocker exec <db> mysqldump … | zstddirectly to the destination storage on the server. Works fine for the same 110GB database, which supports the "control plane vs data plane" fix above.