fix: colocate stub script and data file for preprocess_*_batch dispatch

[GondekNP]

Summary

Fixes the bug diagnosed against a downstream user: preprocessBatch calls never actually reached a remote target. Root cause confirmed against josh main and dev, plus the K8s pod entrypoint script (cloud-img/preprocess-entrypoint.sh):

• GridSpec._render_preprocess_script() wrote its throwaway stub .josh script into bare system /tmp. Harmless for local preprocess() (no staging step), but preprocessBatch derives its MinIO upload scope from the script's parent directory (PreprocessBatchCommand.resolveInputDir()) and requires the data file to resolve as a path inside that same directory (resolveDataFileRelativeToInputDir()) — and stages (uploads) the whole directory before that check runs. A script in /tmp meant every call staged every unrelated file in /tmp (~1500 in the reported case) before failing with dataFile is outside input directory.
• Worse: just isolating the script into its own empty temp dir would only make that failure fast instead of slow — the real data file (e.g. under data/preprocessed/...) still wouldn't be inside it, so the call could never succeed at all. preprocess-entrypoint.sh confirms the remote pod needs both the script and the data file staged from the same directory (find $WORK_DIR -name '*.josh' + $WORK_DIR/$JOSH_DATA_FILE).

Fix

• Adds GridSpec._render_batch_staging_dir(): creates a fresh isolated temp directory containing the rendered stub script plus a symlink to the real data file (not a copy — following batch_orchestrator.assemble_batch_workdir()'s existing convention, so large geospatial files aren't duplicated on disk). All three preprocess_*_batch() methods now stage through it and clean up the whole directory afterward instead of just unlinking the script.
• Fixes JoshCLI.preprocess_batch(): it built CLI args with Path.resolve(), which follows symlinks — silently substituting the symlinked data file's real target path (outside the staging directory) for the path josh actually needs to see. Switched to Path.absolute(), which makes paths absolute without touching symlinks. Verified this distinction concretely against a real symlink before landing on it (resolve() returned the real target's dir, absolute() preserved the staging dir).

Test plan

• 20 new tests across tests/test_cli.py and tests/test_grid.py, including:
  • A dedicated resolve() vs absolute() regression test that fails under the old behavior and passes under the fix.
  • Colocation checks (script.parent == data_file.parent) for all three formats.
  • Symlink correctness: right target, right content, preserved filename/suffix.
  • Minimal upload scope: staging dir contains exactly the script + one data file, even when the real data file has unrelated siblings.
  • Cleanup on success, on a failed-but-not-raised dispatch, and when cli.preprocess_batch() raises (exception still propagates).
  • Independent staging dirs across sequential calls.
  • Contract tests that reimplement josh's own resolveInputDir/resolveDataFileRelativeToInputDir/stageDirectory (Files.walk + isRegularFile) logic in Python and run it against the paths this fix actually produces — since the real validation lives in the JAR and can't be exercised without it.
• Mutation-tested: stashed just the _core.py fix and confirmed 11 of the 12 new GridSpec-level tests fail against the prior code — including reproducing the exact reported symptom (upload-scope walks found 4122 and 2323 files in this container's /tmp, matching the "~1500 unrelated files" report).
• pixi run -e dev pytest tests/ -m "not integration" -q — 1194 passed
• pixi run -e dev ruff check on all changed files — 0 new issues (confirmed via before/after diff against the pre-existing baseline)

Known related limitation (not fixed here)

amend=True for batch dispatch may not fully work: the remote entrypoint always writes fresh to $WORK_DIR/$JOSH_OUTPUT_FILE, and nothing currently stages a pre-existing output .jshd into that path for the JAR to actually amend against. Flagging as a separate, unverified concern — happy to dig into it further if useful.

🤖 Generated with Claude Code