We are moving the AWS Batch sim workloads to a generic runner: the container image holds only the environment, and application code is delivered at submit time as a content-addressed tarball in S3. Design and rationale: ergodicio/continuum-infra#47. Runtime implementation: ergodicio/continuum-infra#61.
The submit-side client currently lives in the infra repo as sim-runner/submit.py (a proof of concept, ~230 lines). It belongs here instead.
Why adept
Every sim repo that would use it already depends on adept — ml-for-lpi, kinetic-srs and vp-turbulence all declare adept @ git+https://github.com/ergodicio/adept.git@main — and adept already carries boto3 plus S3 helpers (adept/utils.py:75, download_from_s3). So a shared client reaches all of them with no new dependency and no new repo. Leaving it in the infra repo means each sim repo either vendors a copy or shells out to a path that only exists on someone's laptop.
Proposed surface
from adept.cloud import submit
job_id = submit(cmd="python run.py --cfg configs/foo", queue="gpu", array=16)
Roughly: bundle the working tree → content-address it → upload if absent → batch.submit_job with the generic job definition → return the job id.
Requirements the proof of concept established
These were found by testing and are easy to get wrong on a rewrite (details in continuum-infra#61):
- Force-include
pyproject.toml and uv.lock regardless of git status. kinetic-srs/.gitignore deliberately ignores uv.lock, so a bundler that merely respects .gitignore ships no lock and the runner has nothing to install from.
- Reproducible archives.
tarfile's w:gz stamps the gzip header with the current time and offers no override, so identical trees hash differently and content addressing never hits. Build uncompressed, gzip with mtime=0, normalise member metadata.
- Exclude committed outputs, loudly. Respecting
.gitignore is not sufficient because these repos commit their outputs — kinetic-srs tracks ~42 MB of plots under sims/, vp-turbulence has ~463 MB of .nc. Drop derived suffixes, but always print the count and size dropped, and offer an escape hatch to put one back.
- Array support.
AWS_BATCH_JOB_ARRAY_INDEX must reach the command, so an N-config scan is one array job rather than N submissions.
Also worth doing here
adept is the natural place to record the resolved ADEPT SHA per cloud run. Both dependent sim repos pin @main, i.e. unpinned, so today a run's ADEPT version is whatever was current when the image was built, with no record of which. Resolving at submit time and shipping the lock inside the bundle fixes that — the run's exact ADEPT commit becomes recoverable from its own artifact bundle.
We are moving the AWS Batch sim workloads to a generic runner: the container image holds only the environment, and application code is delivered at submit time as a content-addressed tarball in S3. Design and rationale: ergodicio/continuum-infra#47. Runtime implementation: ergodicio/continuum-infra#61.
The submit-side client currently lives in the infra repo as
sim-runner/submit.py(a proof of concept, ~230 lines). It belongs here instead.Why adept
Every sim repo that would use it already depends on adept —
ml-for-lpi,kinetic-srsandvp-turbulenceall declareadept @ git+https://github.com/ergodicio/adept.git@main— and adept already carriesboto3plus S3 helpers (adept/utils.py:75,download_from_s3). So a shared client reaches all of them with no new dependency and no new repo. Leaving it in the infra repo means each sim repo either vendors a copy or shells out to a path that only exists on someone's laptop.Proposed surface
Roughly: bundle the working tree → content-address it → upload if absent →
batch.submit_jobwith the generic job definition → return the job id.Requirements the proof of concept established
These were found by testing and are easy to get wrong on a rewrite (details in continuum-infra#61):
pyproject.tomlanduv.lockregardless of git status.kinetic-srs/.gitignoredeliberately ignoresuv.lock, so a bundler that merely respects.gitignoreships no lock and the runner has nothing to install from.tarfile'sw:gzstamps the gzip header with the current time and offers no override, so identical trees hash differently and content addressing never hits. Build uncompressed, gzip withmtime=0, normalise member metadata..gitignoreis not sufficient because these repos commit their outputs — kinetic-srs tracks ~42 MB of plots undersims/, vp-turbulence has ~463 MB of.nc. Drop derived suffixes, but always print the count and size dropped, and offer an escape hatch to put one back.AWS_BATCH_JOB_ARRAY_INDEXmust reach the command, so an N-config scan is one array job rather than N submissions.Also worth doing here
adept is the natural place to record the resolved ADEPT SHA per cloud run. Both dependent sim repos pin
@main, i.e. unpinned, so today a run's ADEPT version is whatever was current when the image was built, with no record of which. Resolving at submit time and shipping the lock inside the bundle fixes that — the run's exact ADEPT commit becomes recoverable from its own artifact bundle.