feat: add an SSH port-forward shadow for air-gapped HPC

[matbun]

Closes #545. Implements the approach discussed in #535.

Problem

The wstunnel shadow assumes the offloaded pod can dial out to a public ingress. Air-gapped HPC sites have neither: compute nodes have no outbound internet access, and the cluster has no publicly reachable ingress. Today the only way to expose a notebook or dashboard running on such a site is a hand-written ~200-line customTemplate per deployment, which is what CERN and BSC both ended up with.

What this does

Adds an SSH port-forward shadow that inverts the direction: instead of the compute node dialing out, the shadow pod dials in to the site's login node and forwards each exposed port to the compute node the job landed on.

browser -> Ingress/Service -> shadow pod (ssh) -> HPC login node -> compute node

The only requirement is outbound SSH from the cluster to the login node. The offloaded pod runs nothing on its side: no wstunnel client, no WireGuard config, no pre-exec injection.

Network:
  EnableTunnel: true
  ShadowMode: ssh
  SSH:
    LoginHost: login.hpc.example.org
    User: alice
    KeySecret: hpc-ssh-key        # or Auth: kerberos + KeytabSecret + Principal

ShadowMode defaults to wstunnel, so existing deployments are unaffected.

Commits

1. refactor: rename wstunnel shadow-pod naming to shadow

The per-pod Deployment/Service is a shadow pod; wstunnel is only one thing that can run inside it, and the SSH shadow runs none. Rendered names change:

	before	after
Deployment/Service	wstunnel-<pod>-<ns>	shadow-<pod>-<ns>
Namespace	<ns>-wstunnel	<ns>-shadow

Go identifiers follow (WstunnelTemplateData -> ShadowTemplateData, createDummyPod -> createShadowPod, ...), and the naming/identity helpers move from mesh.go into a new shadow.go.

Pod annotations (interlink.eu/wstunnel-extra-ports, interlink.virtual-kubelet.io/wstunnel-timeout) and the Wstunnel* config keys are deliberately not renamed — that would break existing manifests for no user-visible gain. Names that genuinely refer to wstunnel keep them.

2. feat: propagate the remote compute node to shadow pods

The shadow is rendered at CreatePod time, before the job is submitted, so nothing in the pod spec can carry the target node. Plugins now report it via a new optional field:

PodStatus.NodeName string `json:"nodeName,omitempty"`

The VK creates a per-shadow ConfigMap <shadow>-node (empty) alongside the shadow and patches the value in once a plugin reports it. A ConfigMap rather than an env var because kubelet refreshes mounted ConfigMaps in place — changing the pod spec would restart the shadow and change its pod IP, which has already been reported to Kubernetes as the offloaded pod's IP.

The value is blanked on shadow creation so a recreated pod can't inherit a dead node, and unchanged values are dropped so the status loop doesn't write on every poll.

omitempty + plugins that never set it = unchanged wire format.

3. feat: add an SSH port-forward shadow for air-gapped HPC

New embedded template templates/shadow-ssh-template.yaml plus a Network.SSH config block. Both auth methods sites actually use are supported: an SSH private key, or a Kerberos keytab with kinit in an init container and a sidecar refreshing the ticket for the life of the tunnel.

A wait-for-node init container polls the node ConfigMap, so a pod whose job is still queued shows Init:0/1 rather than a crash-looping tunnel. Bounded by SSH.NodeWaitTimeout (default 2h — queue waits are normal) with an error that names the actual cause.

Credentials are replicated into the shadow's namespace by default (SSH.ReplicateCredentials), since shadows follow the offloaded pod into arbitrary per-user namespaces with interlink.eu/shadow-same-ns.

ShadowMode: ssh + FullMesh: true is rejected at startup — the mesh needs the compute node to dial the cluster, which is exactly what these sites cannot do. Tracked in #548. UDP ports are skipped with a warning, since ssh -L is TCP only.

The shadow runs docker/Dockerfile.ssh-tunnel (debian-slim + openssh-client, krb5-user, socat), published as ghcr.io/interlink-hq/interlink/ssh-tunnel:<tag> by the existing release workflow. It contains no interLink code. SSH.Image overrides it; the default is pinned to KubeletVersion so image and VK release together.

4. fix: rebuild the SSH shadow tunnel when the job moves to another node

A requeued job comes back on a different host, and the VK republishes it into the node ConfigMap — but ssh already had the old hostname baked into its -L arguments and kept forwarding to the node the job had left.

The container now supervises ssh rather than execing it: it polls the mounted file and rebuilds the forwarders against the new node, in place. An ssh exit with the node unchanged still ends the container, so a dead tunnel is restarted by Kubernetes as before.

5. feat: add an exec forward mode for login nodes that refuse port forwarding

ssh -L needs AllowTcpForwarding on the login node, and a good number of sites do not grant it. SURF's ETP testbed sets AllowTcpForwarding no globally and re-enables it only for the MFA group, so every channel comes back:

channel 1: open failed: administratively prohibited: open failed

Network.SSH.ForwardMode: exec drops the forwarding channel entirely. Each exposed port gets a local socat listener in the shadow, and every accepted connection is relayed by a command run on the login node (Network.SSH.ExecConnectCommand, default nc <compute node> <port>). All it needs is the ability to run a command over SSH — the same privilege the shadow already uses to log in.

    ForwardMode: exec
    ExecConnectCommand: nc     # ncat, or "nc -q 1" on a netcat that does not half-close

The relays share one multiplexed SSH connection, established as an explicit control master before the listeners open. Without it each connection would pay a full handshake — a notebook UI opens dozens — and the login node would see a session per connection.

The two modes are interchangeable above the socket: the Service fronts the same containerPort, and the offloaded workload still just binds 0.0.0.0. portforward stays the default; it is cheaper and has no dependency on the site having a netcat.

6. fix: refuse a compute node name that is not a hostname

The node a plugin reports goes into a shell command twice — the -L argument in portforward mode, and the relay command exec mode sends to the login node — and was neither quoted nor checked. A name containing whitespace splits into an extra ssh argument, and ssh takes options anywhere:

-L 0.0.0.0:8888:h:1 -oProxyCommand=touch:8888 user@login

ProxyCommand runs a command in the shadow container. In exec mode a name containing a quote or $( breaks out of the relay command instead, which the login node's shell then re-parses. For the Slurm plugin this value is hostname -f written to a file in the pod's job directory on a shared filesystem, so it is only as trustworthy as the permissions on that directory.

Refuse anything that is not an RFC 1123 hostname or an IP literal before it reaches the ConfigMap the shadow mounts, and quote it in both modes as well.

7. fix: do not overwrite a credential the shadow namespace already had

Credential replication created-or-updated unconditionally. With interlink.eu/shadow-same-ns the shadow lands in the offloaded pod's own namespace, which on a multi-tenant cluster is somebody's personal namespace, so a Secret that merely shared the configured name was replaced with the HPC key and its contents lost, silently.

Replicated objects now carry interlink.eu/replicated-from and only ones bearing it are overwritten. An unmarked object whose content already matches is adopted rather than refused — that is what a copy made by an earlier version look

Breaking changes

Renaming the rendered resources is a soft break:

• Shadows created before this change keep their old names until their pods are recreated.
• Anything hard-coding a shadow Service name — Ingress, Istio VirtualService, admission policies — needs updating.

Security note

SSH.ReplicateCredentials defaults to true. It copies the SSH key/keytab Secret into each shadow's namespace, which makes it readable by anyone who can read Secrets there. It defaults on because multi-tenant setups (Kubeflow notebooks in per-user namespaces) are unusable otherwise, but it's a real tradeoff — set it to false and provision the Secret yourself if that isn't acceptable for your cluster. The Helm chart only grants the VK cluster-wide Secret write when it's enabled.

Without SSH.KnownHostsConfigMap the shadow uses StrictHostKeyChecking=accept-new. Pinning host keys is documented and recommended for production.

Testing

go build ./... && go vet ./... && go test ./pkg/... clean. Unit tests cover:

• NormalizeShadowConfig defaults and every rejection path (unknown mode, unknown forward mode, ssh+mesh, missing required fields, bad duration)
• both forward modes rendered through the same decoder applyShadowManifests uses, asserting the decoded Deployment — -L per TCP port vs socat listeners plus a control master, UDP skipped, kerberos vs publickey containers/volumes, known_hosts pinning on/off
• node ConfigMap create/blank/patch, no-write when queued, no-write on repeat, cleanup
• credential replication, opt-out, and a clear error when the source Secret is missing

End to end, on real hardware

Three virtual kubelets on a shared Kubernetes cluster, one per site, each covering a different combination:

Site	Auth	Forward mode
SURF ETP (liza.surf.nl)	public key	exec
CERN (slurmgate06)	Kerberos keytab	portforward
BSC MareNostrum5 (alogin1.bsc.es)	public key	portforward

Verified at all three: pod offload (including a GPU pod on SURF), PodStatus.NodeName arriving end to end from the Slurm plugin through the status API into the shadow's ConfigMap, credential replication into the shadow namespace (keytab and krb5.conf for the Kerberos one), and an HTTP service inside the offloaded pod reachable from an ordinary cluster pod through the shadow Service:

$ wget -qO- http://cern-http-interlink-cern-ssh.interlink-cern-ssh-shadow.svc.cluster.local:18080/
hello from hpc-cloud-test001.cern.ch at 2026-07-31T14:04:35+02:00

Also verified:

• The queued case, which is the normal one. A BSC job sat in PENDING for ~20 minutes while the shadow showed Init:0/1 with an empty ConfigMap, then came up on its own once Slurm allocated a node.
• The requeue case (commit 4), by republishing a different node into the ConfigMap: the shadow logged compute node changed from ... to ..., rebuilding the tunnel and kept serving, with container restartCount still 0 — so the pod IP already reported for the offloaded pod did not change.
• A JupyterLab session on every site, driven from an ordinary cluster pod through the shadow Service: /lab returns 200 with <title>JupyterLab</title>, POST /api/kernels starts a Python kernel on the compute node, and a file round-trips through the contents API. 45-106 ms per request, including the exec-mode site where every request crosses an SSH channel and a nc hop.
• The node-name check (commit 6), by writing h -oProxyCommand=touch /tmp/pwned into the shared-filesystem file the plugin reads. The virtual kubelet refused it, the ConfigMap kept the real node, the tunnel kept serving, and nothing ran in the shadow container.

Related

• Helm chart support: feat: expose the SSH port-forward shadow (#546) interlink-helm-chart#56
• Plugin side (SLURM PodStatus.NodeName reporting): feat: report the allocated compute node in pod status interlink-slurm-plugin#148 — depends on this being released, needs a go.mod bump afterwards
• Docs: docs: add an SSH tunnel configuration guide #551
• Mesh over SSH: out of scope, Explore the ssh-only deployment with mesh #548

[netlify]

✅ Deploy Preview for interlink-dev canceled.

Name	Link
🔨 Latest commit	04020c1
🔍 Latest deploy log	https://app.netlify.com/projects/interlink-dev/deploys/6a6cb93afd44fb0008e5c9a4