Blob staging and commit paths allow unbounded memory and disk exhaustion

[Bnjoroge1]

Severity

P1 / high availability and resource-exhaustion

Review baseline

0bd0e314696e520aa47620cbe8f008d010e01e57

Summary

The Azure-compatible blob upload/download implementation accepts very large request bodies and performs block assembly/downloads with full-buffer reads. There is no aggregate object quota, per-job quota, block-count cap, or staging-GC policy.

A malicious or buggy authenticated job can therefore fill host disk or cause the control-plane process to allocate very large buffers.

Source observations

• Blob upload routes accept request bodies up to 512 MiB per request.
• Staged block uploads are written under the server state directory.
• Block-list commit reads each block and appends it into one in-memory Vec<u8> before writing the final object.
• Blob downloads use whole-file reads rather than streaming.
• There is no aggregate blob-size limit, block-count limit, per-job/repo storage quota, or cleanup policy for abandoned staged blocks.

Attack/failure modes

1. Stage many maximum-sized blocks and never commit them -> fill local disk.
2. Commit a block list whose total size exceeds available RAM -> OOM the server while assembling the final object.
3. Run several large commits/downloads concurrently -> amplify resident memory and I/O pressure.
4. Leave failed/abandoned staging directories indefinitely -> persistent disk leak.

Impact

• Control-plane OOM and restart loops.
• Host disk exhaustion affecting unrelated jobs.
• Long I/O stalls on the shared state volume.
• Cross-tenant denial of service in multi-tenant deployments.

Recommended remediation

• Mint upload capabilities bound to {job, object-kind, object-id, max-size, expiry}.
• Enforce maximum object size and maximum block count before accepting/committing a block list.
• Stream block assembly into an atomic temporary file rather than accumulating the full object in memory.
• Stream downloads from disk/object storage instead of read()ing the full file.
• Add per-job, per-repository, per-tenant and global disk quotas.
• Garbage-collect stale staging directories after an explicit TTL.
• Record metrics for staged bytes, committed bytes, rejected uploads, GC, and active assembly memory.

Acceptance criteria

• A job cannot stage or commit more than its configured quota.
• Commit memory usage remains bounded independently of final artifact/cache size.
• Downloads are streamed.
• Abandoned block directories are automatically reclaimed.
• Concurrent large uploads cannot exhaust process memory or the state filesystem.