Skip to content

Fix multi-tile staging gaps and add Slurm output collection#113

Open
mak8427 wants to merge 9 commits into
ESAOpenSR:mainfrom
mak8427:main
Open

Fix multi-tile staging gaps and add Slurm output collection#113
mak8427 wants to merge 9 commits into
ESAOpenSR:mainfrom
mak8427:main

Conversation

@mak8427
Copy link
Copy Markdown
Contributor

@mak8427 mak8427 commented May 27, 2026

Description

This PR improves the SRGAN HPC deployment workflow for large Sentinel-2 runs.

It fixes staging gaps near Sentinel-2 tile boundaries by keeping all candidate STAC items returned for a patch and passing them to Cubo. The existing mosaic_valid logic
can then build a valid-pixel union across neighboring tiles instead of using only the first STAC item. so that multiple tiles will be merged together to reach full coverage instead of having holes overe big areas

It also adds an automatic Slurm follow-up collection job for grid/AOI array submissions. After the array completes successfully, a dependent collect job organizes patch
outputs into the run’s collected/ folder. Collection now moves GeoTIFF outputs instead of copying them, so results are stored only in the organized output location.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

How Has This Been Tested?

  • Ran deployment test suite:

pytest tests/test_deployment -q

Result:

30 passed

  • Validated the reusable SRGAN HPC runtime config:

python -m deployment.srgan_hpc.cli validate-config --config configs/runtime.srgan_base.yaml

  • Ran a dry-run grid submission and confirmed the summary reports the follow-up collect job.
  • Verified on the cluster that a previously problematic tile-boundary patch now stages with multiple STAC candidates and reaches full valid coverage:

candidates=2 tiles=32UPB,32UPC
full=1.000 center=1.000

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant