Make the K8s batch target ready to test on Nautilus/NRP

[GondekNP]

Groundwork for testing batchRemote/preprocessBatch on Nautilus, plus one credential-hygiene bug found while reading the dispatch path. The K8s target has only ever run against kind (CI) and GKE Autopilot, though README.md and llms-full.txt already advertise Nautilus support.

1. The credential Secret was never cleaned up

Every dispatch created a josh-creds-<jobId> Secret holding live object-store credentials, and nothing deleted it. ttlSecondsAfterFinished reaps the Job and its pods but not the Secret, so a shared namespace accumulated working credentials indefinitely — and because both K8s targets carried their own copy of the Secret code, the leak existed twice.

Fixed by adding an ownerReferences entry pointing at the created Job, so K8s garbage collection removes the Secret with it. Two ordering details worth a reviewer's attention:

• The Secret is still created before the Job. Creating the Job first lets a pod be scheduled against a secretKeyRef that does not exist yet, landing the container in CreateContainerConfigError — which KubernetesPollingStrategy reports as a job failure. There's a regression test pinning the order.
• The owner reference is therefore added after the Job exists, since it needs the server-assigned UID. Binding failures warn rather than throw: the API server has already accepted the Job at that point, so a namespace withholding patch rights on Secrets degrades to the old behaviour instead of failing a live dispatch.

Shared logic extracted into KubernetesJobSecret so both dispatch flows clean up identically.

2. Nautilus/NRP target profile

examples/test/nautilus/nautilus.json — fill in kubernetes.namespace and minio_bucket and it's ready. It is deliberately not a copy of the GKE profile:

• NRP requires limits within 20% of requests. The GKE profile's 2Gi request / 4Gi limit violates that. Requests and limits are equal here, and include ephemeral-storage, since staging writes inputs to the pod's ephemeral disk and a job with large .jshd inputs gets evicted without a request for it.
• Pods use the in-cluster Ceph RGW (http://rook-ceph-rgw-nautiluss3.rook) while the client uses the public gateway (https://s3-west.nrp-nautilus.io). This is exactly what pod_minio_endpoint was built for — the in-cluster path talks to the OSDs directly for higher bandwidth.
• No "spot". applySpotConfig emits a cloud.google.com/gke-spot selector and toleration; on NRP that just makes pods unschedulable.

A loader test covers the committed profile so a typo or schema change doesn't reach whoever runs the first NRP job.

3. DNS probe followed a hardcoded GCS hostname

Both entrypoints waited for storage.googleapis.com to resolve before staging. The probe is worth having — a pod's resolver can be briefly unusable right after start, and losing stageFromMinio costs a full JVM startup — but resolving a GCS hostname says nothing about reaching an in-cluster MinIO or NRP's Ceph, and on a cluster with no route to it the loop burns 20s per pod for no signal. Now parses the host out of $MINIO_ENDPOINT, which pods already receive from the credential Secret.

Separate commit, so it can be dropped independently if you'd rather not touch the image in this PR.

Still unverified

Recorded as a checklist in BATCH_REMOTE.md. The one that could be structural rather than cosmetic: the batch image is bare eclipse-temurin:21-jre and runs as root with no securityContext, so a namespace enforcing Pod Security restricted would reject the pod at admission and no profile setting could rescue it. NRP's published policies don't mention PSA, so this needs an empirical check rather than a code change up front.

Related gaps I did not touch, since they're beyond this scope: no serviceAccountName (blocks workload identity — forces static keys, fine on NRP but not for a keyless EKS/GKE partner), no imagePullSecrets, no generic tolerations, and backoffLimit hardcoded at 3.

Testing

Full suite green. New coverage: Secret→Job ordering, owner reference contents, graceful degradation when binding is refused (both targets), and the committed profile parsing.

🤖 Generated with Claude Code

[GondekNP]

Note on deploy ordering, since the two fixes land through different paths:

• The Secret ownership fix is client-side — effective as soon as you build the jar, no image involved.
• The entrypoint DNS probe is baked into the image, and buildBatchImage is gated to main/dev/feat/k8s-batch (build.yaml:698). So ghcr.io/schmidtdse/josh/joshsim-batch:latest — which the Nautilus profile references — keeps the old GCS-hardcoded probe until this merges to dev.

For Nautilus specifically that gap is close to harmless: NRP nodes have public egress, so storage.googleapis.com resolves immediately and the old probe returns without stalling. It only bites on a cluster with no route out, where the old probe burns its full 20s for no signal. Worth knowing so a first-run stall on a restricted cluster is not misread as something else.

CI note: the kind integration test builds the image locally from the Dockerfile, so the new entrypoint was exercised here — it just is not published yet.

🤖 Generated with Claude Code

[GondekNP]

Validated on Nautilus. All five checklist items pass on namespace schmidtdse, against the committed profile unchanged apart from the two documented substitutions.

Test	Result
N1 smoke test (1 replicate)	PASS — 40s end to end
N2 fan-out (3 replicates)	PASS — indices 0/1/2, three distinct {replicate} CSVs
N3 preprocessing round-trip	PASS — GeoTIFF → 16MB .jshd downloaded
N4 Secret GC'd with its Job	PASS — see below
N5 pod admission	PASS — no PSA label on the namespace, root image admitted

N4 is the one worth calling out, since it is this PR's actual fix verified against a real API server rather than mocks or kind. Both dispatch paths bound their Secret:

josh-creds-6d0abea7-...  -> Job/josh-6d0abea7-... controller=True   (preprocess)
josh-creds-eb936733-...  -> Job/josh-eb936733-... controller=True   (run)

Deleting the Job removed the Secret with no manual cleanup — the cascade works. Zero could not bind Secret warnings, so the binding took the real path rather than degrading to warn-and-leak.

N5 resolves the open question in the PR body. The namespace carries no pod-security.kubernetes.io/* label, so the root-running image is admitted and the securityContext gap I flagged is not a blocker on NRP. It remains a real gap for any cluster that does enforce restricted.

Two findings beyond the checklist:

Pods scheduled across institutions. The three N2 replicates landed on nodes at SDSC, SDSC-HaoSu and UC Santa Cruz, all writing back to the west Ceph pool through the in-cluster RGW service. The in-cluster endpoint resolves from every site, so pod_minio_endpoint needs no per-site variation.

Auth is interactive, and that is the real limit on automation. NRP has no static credential in the kubeconfig — it execs kubectl oidc-login (kubelogin), which must be installed as kubectl-oidc_login on PATH. Fabric8 shells out to the same plugin, so josh inherits the requirement. First use is a browser round-trip; offline_access then yields a refresh token so renewals are silent. Unattended dispatch therefore needs a pre-warmed token cache plus the plugin binary, so for CI or scheduled runs a namespace ServiceAccount token is the better answer. Recorded in BATCH_REMOTE.md; worth a follow-up.

The DNS probe commit is also load-bearing for this config: pods stage from http://rook-ceph-rgw-nautiluss3.rook, which the old hardcoded storage.googleapis.com probe told us nothing about.

🤖 Generated with Claude Code