From 98f971890e8aa1bddb2587bab82849cfc2070339 Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Wed, 29 Jul 2026 18:51:46 +0200
Subject: [PATCH 1/5] docs: add an SSH tunnel configuration guide

Documents the ssh shadow mode: what it does, how it differs from wstunnel
and full mesh, both authentication methods, host key pinning, how the
compute node reaches the shadow, and the credential replication tradeoff.

Kept to its own guide rather than folded into the wstunnel one, since the
two are alternatives rather than variations. The cookbook rework is
tracked separately in interlink-hq/interLink#547.

Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .../guides/14-ssh-tunnel-configuration.mdx    | 281 ++++++++++++++++++
 1 file changed, 281 insertions(+)
 create mode 100644 docs/docs/guides/14-ssh-tunnel-configuration.mdx

diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
new file mode 100644
index 00000000..73694126
--- /dev/null
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -0,0 +1,281 @@
+---
+title: "SSH Tunnel Configuration"
+description: "Reaching services inside offloaded pods on air-gapped HPC sites over an SSH login node"
+sidebar_position: 14
+---
+
+# SSH Tunnel Configuration
+
+Many HPC sites give compute nodes no outbound internet access, and many Kubernetes
+clusters have no publicly reachable ingress. The [wstunnel](./10-wstunnel-configuration.mdx)
+shadow needs both: the workload dials out to an ingress and runs a wstunnel client
+there. On an air-gapped site it cannot.
+
+The SSH shadow inverts the direction. Instead of the compute node dialing out, the
+cluster dials **in** to the site's SSH login node and forwards each exposed port
+from the compute node the job landed on:
+
+```
+browser → Ingress / Service → shadow pod (ssh -L) → HPC login node → compute node (Jupyter)
+```
+
+The only requirement is outbound SSH from the cluster to the login node. The
+offloaded pod runs nothing on its side: no wstunnel client, no WireGuard
+configuration, no pre-exec injection.
+
+:::info
+The SSH shadow replaces **wstunnel**, not [full mesh](./13-mesh-network-configuration.mdx).
+It exposes the offloaded pod's ports to the cluster; it does not give the pod access
+back into the cluster, so an offloaded workload still cannot reach in-cluster object
+storage or message buses. Combining the two is rejected at startup and tracked in
+[#548](https://github.com/interlink-hq/interLink/issues/548).
+:::
+
+---
+
+## Which shadow does what
+
+| | Direction | Trigger | Workload must run |
+| --- | --- | --- | --- |
+| `wstunnel` (default) | cluster → pod | `EnableTunnel` + exposed ports | wstunnel client, dials out |
+| `ssh` | cluster → pod | `EnableTunnel` + exposed ports | nothing |
+| full mesh | bidirectional | `FullMesh: true`, every pod | `mesh.sh` (slirp4netns + WireGuard) |
+
+---
+
+## Configuration
+
+### Virtual Kubelet
+
+```yaml
+# VirtualKubeletConfig.yaml
+Network:
+  EnableTunnel: true
+  ShadowMode: ssh
+  SSH:
+    LoginHost: login.hpc.example.org
+    User: alice
+    KeySecret: hpc-ssh-key
+```
+
+`EnableTunnel` turns shadow pods on for offloaded pods with exposed ports;
+`ShadowMode` picks which shadow is rendered. Both are needed.
+
+### Helm
+
+```yaml
+virtualNode:
+  network:
+    enableTunnel: true
+    shadowMode: ssh
+    ssh:
+      loginHost: login.hpc.example.org
+      user: alice
+      keySecret: hpc-ssh-key
+```
+
+See [`examples/ssh_tunnel.yaml`](https://github.com/interlink-hq/interlink-helm-chart/blob/main/interlink/examples/ssh_tunnel.yaml)
+in the chart repository for a complete deployment.
+
+### Options
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `LoginHost` | — | SSH login node to forward through (**required**) |
+| `User` | — | Login name on that node (**required**) |
+| `Port` | `22` | Login node's SSH port |
+| `Image` | `ghcr.io/interlink-hq/interlink/ssh-tunnel:<version>` | Image the shadow runs. Needs an ssh client, plus `kinit` for Kerberos |
+| `Auth` | `publickey` | `publickey` or `kerberos` |
+| `KeySecret` | — | Secret holding the private key (`publickey`) |
+| `KeySecretKey` | `id_ed25519` | Key inside `KeySecret` |
+| `KeytabSecret` | — | Secret holding the keytab (`kerberos`) |
+| `KeytabSecretKey` | `user.keytab` | Key inside `KeytabSecret` |
+| `Principal` | — | Kerberos principal (`kerberos`) |
+| `Krb5ConfigMap` | — | ConfigMap with a `krb5.conf`, mounted at `/etc/krb5.conf` |
+| `KnownHostsConfigMap` | — | ConfigMap with a `known_hosts` file |
+| `ReplicateCredentials` | `true` | Copy the credential into each shadow's namespace |
+| `NodeWaitTimeout` | `2h` | How long the shadow waits for the compute node |
+| `ExtraOptions` | `[]` | Extra ssh options, each passed as `-o <option>` |
+
+Misconfiguration fails at **startup**, not on the first offloaded pod, so a missing
+`LoginHost` or a `kerberos` block without a `Principal` is caught immediately.
+
+---
+
+## Authentication
+
+### Public key
+
+```bash
+ssh-keygen -t ed25519 -f ./hpc_key -N ''
+ssh-copy-id -i ./hpc_key.pub alice@login.hpc.example.org
+
+kubectl create secret generic hpc-ssh-key -n interlink \
+  --from-file=id_ed25519=./hpc_key
+```
+
+```yaml
+SSH:
+  Auth: publickey
+  KeySecret: hpc-ssh-key
+```
+
+### Kerberos
+
+For GSSAPI sites (`GSSAPIAuthentication yes` on the login node):
+
+```bash
+kubectl create secret generic hpc-keytab -n interlink \
+  --from-file=user.keytab=/path/to/user.keytab
+```
+
+```yaml
+SSH:
+  Auth: kerberos
+  KeytabSecret: hpc-keytab
+  Principal: alice@EXAMPLE.ORG
+  Krb5ConfigMap: krb5-config   # if the image's /etc/krb5.conf lacks your realm
+```
+
+The shadow runs `kinit` in an init container and refreshes the ticket from a
+sidecar for the life of the tunnel, so long-running notebooks survive ticket
+expiry.
+
+### Host key verification
+
+Without `KnownHostsConfigMap` the shadow uses `StrictHostKeyChecking=accept-new`,
+which trusts whatever key the login node presents on first contact. That is a small
+MITM window on every new shadow pod. For production, pin the key:
+
+```bash
+ssh-keyscan login.hpc.example.org > known_hosts
+kubectl create configmap hpc-known-hosts -n interlink --from-file=known_hosts
+```
+
+```yaml
+SSH:
+  KnownHostsConfigMap: hpc-known-hosts
+```
+
+This switches the shadow to `StrictHostKeyChecking=yes`.
+
+---
+
+## How the compute node is discovered
+
+The shadow is created when the pod is, which is long before the batch system has
+scheduled anything — so at creation time there is no node to forward to yet.
+
+1. interLink creates a ConfigMap, `<shadow>-node`, holding an empty `compute-node` key.
+2. The plugin reports the allocated node as `PodStatus.NodeName` once the job runs.
+3. interLink patches that value into the ConfigMap. It is mounted, not injected into
+   the pod spec, so kubelet refreshes it in place — restarting the shadow would
+   change its pod IP, which Kubernetes has already been told is the offloaded pod's IP.
+4. The shadow's `wait-for-node` init container polls the mounted file, then
+   `ssh-forward` starts the tunnel.
+
+While the job is queued the shadow sits in `Init:0/1`. That is the expected state,
+not a failure — it reads as "waiting for the allocation" rather than a crash-looping
+tunnel. After `NodeWaitTimeout` the init container fails with an explicit message.
+
+:::warning
+This requires a plugin that reports `PodStatus.NodeName`. A plugin that does not
+will leave the shadow waiting until the timeout expires.
+:::
+
+---
+
+## Credential replication
+
+Shadow pods follow the offloaded pod. With `interlink.eu/shadow-same-ns: "true"` — the
+usual choice for multi-tenant setups such as Kubeflow notebooks — that means the
+shadow lands in a per-user namespace, where the SSH credential does not exist.
+
+By default interLink copies the credential Secret, and any `Krb5ConfigMap` /
+`KnownHostsConfigMap`, from its own namespace into the shadow's namespace.
+
+:::warning
+Replication makes the credential readable by anyone who can read Secrets in those
+namespaces. If that is not acceptable, set `ReplicateCredentials: false` and
+provision the Secret in each namespace yourself, through a secrets operator or a
+replication controller.
+:::
+
+---
+
+## Pod configuration
+
+Nothing special is required — an offloaded pod that exposes a port gets a shadow:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: notebook
+  annotations:
+    # Keep the shadow in this pod's namespace, so it can mount credentials
+    # replicated here rather than into a separate <ns>-shadow namespace.
+    interlink.eu/shadow-same-ns: "true"
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: hpc-vk
+  tolerations:
+  - key: virtual-node.interlink/no-schedule
+    operator: Exists
+    effect: NoSchedule
+  containers:
+  - name: jupyter
+    image: docker://jupyter/scipy-notebook
+    ports:
+    - containerPort: 8888
+      name: notebook
+```
+
+The shadow's Service carries the same ports, so anything that can reach the Service —
+an Ingress, an Istio VirtualService, `kubectl port-forward` — reaches the notebook.
+
+`interlink.eu/wstunnel-extra-ports` works here too, for ports not declared in the
+container spec.
+
+:::note
+`ssh -L` forwards TCP only. UDP ports are skipped, with a warning in the virtual
+kubelet log.
+:::
+
+---
+
+## Troubleshooting
+
+**Shadow stuck in `Init:0/1`.** Normal while the job is queued. If it persists,
+check that the job is actually running and that the plugin reports the node:
+
+```bash
+kubectl get configmap <shadow>-node -o jsonpath='{.data.compute-node}'
+```
+
+Empty means no plugin has reported one yet.
+
+**`ssh-forward` crash-loops with a permission error.** The credential Secret is
+missing from the shadow's namespace, or `KeySecretKey` does not match the key inside
+it. With `ReplicateCredentials: false` the Secret must already exist there.
+
+**Connection refused through the tunnel.** The tunnel reached the compute node but
+nothing is listening on that port. Services often bind `127.0.0.1` by default; bind
+`0.0.0.0` instead — for a notebook, `--ip=0.0.0.0`.
+
+**Host key verification failed.** `KnownHostsConfigMap` does not contain the login
+node's current key. Re-run `ssh-keyscan` and update the ConfigMap.
+
+**An nginx wstunnel Deployment appears instead of the SSH shadow.** A custom template
+file is overriding the built-in one. The virtual kubelet prefers
+`Network.WstunnelTemplatePath` over its own templates; the Helm chart stops mounting
+it in `ssh` mode, but a hand-written config may still point at one.
+
+---
+
+## References
+
+- [Discussion #535](https://github.com/interlink-hq/interLink/discussions/535) — the original proposal
+- [Issue #545](https://github.com/interlink-hq/interLink/issues/545) — in-cluster SSH deployment
+- [Wstunnel Configuration](./10-wstunnel-configuration.mdx)
+- [Mesh Network Configuration](./13-mesh-network-configuration.mdx)

From e0433b4f445ab52b70b83855736a3989a3a507ab Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Wed, 29 Jul 2026 20:31:08 +0200
Subject: [PATCH 2/5] docs: collapse the cookbook to edge and in-cluster (#547)

The cookbook offered three deployment scenarios, but "tunneled" was never
a third place to deploy interLink - it is an in-cluster deployment where
the plugin happens to sit on the remote system and the API server reaches
it over an SSH tunnel. The API server is in the cluster either way.

So the cookbook now has two pages, and the in-cluster one covers both
variants:

  - plugin in the cluster, reaching the batch system itself (shared
    filesystem plus SSH shims, or a remote API)
  - plugin on the remote system, reached through the ssh-tunnel binary
    over a Unix socket

3-tunneled.mdx is removed and intro.mdx updated to match, including the
dead link to it.

Most of that page was systemd boilerplate already covered verbatim by the
systemd guide - service user creation, log rotation, health checks,
service management. Dropped, rather than duplicated a third time. What
was genuinely tunnel-specific moves with it: the ssh-tunnel unit and its
ordering constraint now live in the systemd guide, and the authorized_keys
and sshd_config restrictions stay on the cookbook page.

Two things worth flagging for review:

The in-cluster page now mentions SSH twice, for unrelated reasons: the
ssh-tunnel binary carries interLink's control plane, while the SSH shadow
carries traffic into an already-running offloaded pod. There is an admonition
drawing that line, because conflating them would be easy and expensive.

The page also ends with a short comparison of the three ways to reach a
service inside an offloaded pod - wstunnel, ssh, full mesh - and what each
one needs from the site, pointing at the respective guides rather than
explaining them again.

Cookbook sidebar_position values were all 3, so the pages were ordered by
filename; edge and in-cluster are now 1 and 2.

Docs build passes with onBrokenLinks: throw.

Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 docs/docs/cookbook/1-edge.mdx              |   2 +-
 docs/docs/cookbook/2-incluster.mdx         | 251 ++++++-
 docs/docs/cookbook/3-tunneled.mdx          | 735 ---------------------
 docs/docs/guides/08-systemd-deployment.mdx |  63 ++
 docs/docs/intro.mdx                        |  40 +-
 5 files changed, 322 insertions(+), 769 deletions(-)
 delete mode 100644 docs/docs/cookbook/3-tunneled.mdx

diff --git a/docs/docs/cookbook/1-edge.mdx b/docs/docs/cookbook/1-edge.mdx
index 23d0739a..3f15f440 100644
--- a/docs/docs/cookbook/1-edge.mdx
+++ b/docs/docs/cookbook/1-edge.mdx
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 3
+sidebar_position: 1
 ---
 
 import Tabs from "@theme/Tabs";
diff --git a/docs/docs/cookbook/2-incluster.mdx b/docs/docs/cookbook/2-incluster.mdx
index ac396986..f4bb24b2 100644
--- a/docs/docs/cookbook/2-incluster.mdx
+++ b/docs/docs/cookbook/2-incluster.mdx
@@ -1,5 +1,5 @@
 ---
-sidebar_position: 3
+sidebar_position: 2
 ---
 
 import Tabs from "@theme/Tabs";
@@ -9,7 +9,9 @@ import useBaseUrl from "@docusaurus/useBaseUrl";
 
 # In-cluster deployment
 
-Deploy interLink in the local K8S cluster.
+The interLink API server runs inside your Kubernetes cluster, next to the virtual
+kubelet. Nothing has to be installed on an edge node, and no interLink component
+is exposed to the internet.
 
 <ThemedImage
   alt="Docusaurus themed image"
@@ -19,13 +21,27 @@ Deploy interLink in the local K8S cluster.
   }}
 />
 
-## Install interLink
+The remaining choice is where the **plugin** runs, and how the API server reaches
+it:
+
+| | Plugin runs | API to plugin link | Use when |
+| --- | --- | --- | --- |
+| [In the cluster](#plugin-in-the-cluster) | as a container in the same pod | localhost | the plugin can reach the remote system on its own — a shared filesystem, an SSH shim, a REST API |
+| [On the remote system](#plugin-on-the-remote-system) | on the login node or an edge host | SSH tunnel over a Unix socket | the plugin has to run where the batch system is, and you cannot expose a port for it |
+
+If instead you want the API server *and* the plugin to run on the remote side, see
+the [edge node deployment](./1-edge.mdx).
+
+---
+
+## Plugin in the cluster
+
+Everything runs in one pod: virtual kubelet, interLink API server and plugin.
 
 ### Deploy Kubernetes components
 
-The deployment of the Kubernetes components are managed by the official
-[HELM chart](https://github.com/interlink-hq/interlink-helm-chart). Depending on
-the scenario you selected, there might be additional operations to be done.
+The deployment of the Kubernetes components is managed by the official
+[HELM chart](https://github.com/interlink-hq/interlink-helm-chart).
 
 - Create an helm values file:
 
@@ -127,6 +143,205 @@ You can find a demo pod to test your setup
 To start debugging in case of problems we suggest starting from the pod
 containers logs!
 
+### Reaching the batch system from inside the cluster
+
+The plugin still has to submit jobs somewhere. Which mechanism it uses is a
+plugin concern rather than an interLink one, but the common ones are:
+
+- **Shared filesystem plus SSH shims.** Mount the remote scratch area into the
+  plugin container, and point the plugin's `SbatchPath` / `SqueuePath` /
+  `ScancelPath` at small wrappers that `exec ssh user@login /usr/bin/<cmd>`. The
+  plugin never has to know it is not running on the login node.
+- **A plugin that speaks a remote API** — Kubernetes, a cloud batch service, a
+  site REST endpoint.
+
+If neither fits, run the plugin on the remote system instead.
+
+---
+
+## Plugin on the remote system
+
+Some sites will not let you expose a port for the plugin, but do allow outbound
+SSH. In that case the plugin runs on the login node and the API server reaches it
+through an SSH tunnel terminating on a local Unix socket.
+
+<ThemedImage
+  alt="Docusaurus themed image"
+  sources={{
+    light: useBaseUrl("/img/scenario-3_light.svg"),
+    dark: useBaseUrl("/img/scenario-3_dark.svg"),
+  }}
+/>
+
+```
+[Virtual Kubelet] -> [interLink API] -> [Unix socket] -> [SSH tunnel] -> [Plugin]
+     (local)             (local)          (local)        (ssh bridge)   (remote)
+```
+
+:::info
+
+This tunnel carries interLink's **control plane** — the API server talking to its
+plugin. It is unrelated to the
+[SSH shadow](../guides/14-ssh-tunnel-configuration.mdx), which carries traffic
+*into* an already-running offloaded pod. A deployment can use either, both, or
+neither.
+
+:::
+
+### Prerequisites
+
+1. **SSH access** to the remote system where the plugin runs
+2. **SSH key pair** for authentication
+3. **Network connectivity** from the local system to the remote SSH server
+4. **interLink binary** built with the ssh-tunnel command (`make ssh-tunnel`)
+
+#### SSH key setup
+
+```bash
+# Generate SSH key pair
+ssh-keygen -t rsa -b 4096 -f ~/.ssh/interlink_rsa
+
+# Copy public key to remote server
+ssh-copy-id -i ~/.ssh/interlink_rsa.pub user@remote-server
+
+# Test SSH connection
+ssh -i ~/.ssh/interlink_rsa user@remote-server
+```
+
+#### Optional: host key verification
+
+```bash
+# Extract host public key from remote server
+ssh-keyscan -t rsa remote-server > ~/.ssh/interlink_host_key
+
+# Or get it from known_hosts
+ssh-keygen -F remote-server -f ~/.ssh/known_hosts | grep -o 'ssh-rsa.*' > ~/.ssh/interlink_host_key
+```
+
+### Step 1: point the API server at a Unix socket
+
+```yaml title="InterLinkConfig.yaml"
+# Use Unix socket for local communication
+InterlinkAddress: "unix:///tmp/interlink.sock"
+InterlinkPort: ""  # Not used for Unix sockets
+
+# Remote plugin configuration
+SidecarURL: "http://remote-plugin"
+SidecarPort: "4000"
+
+VerboseLogging: true
+ErrorsOnlyLogging: false
+DataRootFolder: "/tmp/interlink"
+```
+
+### Step 2: point the virtual kubelet at the same socket
+
+```yaml title="VirtualKubeletConfig.yaml"
+InterlinkURL: "unix:///tmp/interlink.sock"
+InterlinkPort: ""  # Not used for Unix sockets
+
+VerboseLogging: true
+ErrorsOnlyLogging: false
+
+NodeName: "my-interlink-node"
+NodeLabels:
+  "interlink.cern.ch/provider": "remote-hpc"
+```
+
+### Step 3: start the tunnel
+
+```bash
+./bin/ssh-tunnel \
+  -addr "remote-server:22" \
+  -user "username" \
+  -keyfile "~/.ssh/interlink_rsa" \
+  -lsock "/tmp/interlink.sock" \
+  -rport "4000" \
+  -hostkeyfile "~/.ssh/interlink_host_key"   # optional, but recommended
+```
+
+| Option | Description | Required |
+| --- | --- | --- |
+| `-addr` | SSH server address as `hostname:port` | Yes |
+| `-user` | Username for SSH authentication | Yes |
+| `-keyfile` | Path to private key file | Yes |
+| `-lsock` | Path to local Unix socket | Yes |
+| `-rport` | Remote port where the plugin listens | Yes |
+| `-hostkeyfile` | Path to host public key for verification | No |
+
+Start the components in dependency order: tunnel, then API server, then virtual
+kubelet.
+
+:::note
+
+To run these as managed services, see the
+[systemd deployment guide](../guides/08-systemd-deployment.mdx), which includes
+the SSH tunnel unit and the ordering constraints between the three.
+
+:::
+
+### Hardening the tunnel account
+
+Restrict what the tunnel key is allowed to do on the remote side:
+
+```bash title="~/.ssh/authorized_keys (remote)"
+command="/usr/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding ssh-rsa AAAAB3... interlink-tunnel-key
+```
+
+```bash title="/etc/ssh/sshd_config.d/interlink.conf"
+Match User interlink
+    AllowTcpForwarding yes
+    AllowStreamLocalForwarding yes
+    PermitTunnel no
+    X11Forwarding no
+    AllowAgentForwarding no
+    PermitTTY no
+    ForceCommand /bin/false
+```
+
+### Troubleshooting
+
+```bash
+# Is the tunnel process alive?
+ps aux | grep ssh-tunnel
+
+# Does the socket answer?
+curl -s --unix-socket /tmp/interlink.sock http://unix/pinglink
+
+# Is the plugin listening on the remote side?
+ssh user@remote-server 'netstat -tlnp | grep :4000'
+```
+
+---
+
+## Verify the setup
+
+```bash
+# Check if node appears in Kubernetes
+kubectl get nodes
+
+# Deploy a test pod
+kubectl apply -f - <<EOF
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-incluster
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: interlink-in-cluster
+  tolerations:
+    - key: virtual-node.interlink/no-schedule
+      operator: Exists
+  containers:
+  - name: test
+    image: busybox
+    command: ["sleep", "3600"]
+EOF
+
+kubectl get pod test-incluster -o wide
+kubectl logs test-incluster
+```
+
 :::note
 
 When the virtual node registers, the virtual kubelet requests a
@@ -142,3 +357,27 @@ kubectl certificate approve <csr-name>
 ```
 
 :::
+
+---
+
+## Reaching services inside offloaded pods
+
+Everything above gets *jobs* onto the remote system. Reaching a service that runs
+inside an offloaded pod — a notebook, a dashboard — is a separate concern, handled
+by a shadow pod that interLink creates for any offloaded pod with exposed ports.
+
+| | Direction | Requires | Guide |
+| --- | --- | --- | --- |
+| wstunnel | cluster to pod | outbound internet from the compute node, and a public ingress on the cluster | [Wstunnel](../guides/10-wstunnel-configuration.mdx) |
+| SSH | cluster to pod | outbound SSH from the cluster to a login node | [SSH tunnel](../guides/14-ssh-tunnel-configuration.mdx) |
+| Full mesh | bidirectional | same as wstunnel, plus an unprivileged network namespace on the compute node | [Mesh network](../guides/13-mesh-network-configuration.mdx) |
+
+Air-gapped sites — compute nodes with no route out, clusters with no public
+ingress — generally want the SSH one.
+
+:::note
+
+For additional case studies and advanced configurations, reach out to the
+interLink community through the Slack channel.
+
+:::
diff --git a/docs/docs/cookbook/3-tunneled.mdx b/docs/docs/cookbook/3-tunneled.mdx
deleted file mode 100644
index a891069c..00000000
--- a/docs/docs/cookbook/3-tunneled.mdx
+++ /dev/null
@@ -1,735 +0,0 @@
----
-sidebar_position: 3
----
-
-import Tabs from "@theme/Tabs";
-import TabItem from "@theme/TabItem";
-import ThemedImage from "@theme/ThemedImage";
-import useBaseUrl from "@docusaurus/useBaseUrl";
-
-# Tunneled deployment
-
-Deploy interLink components in both systems, linked through a tunnelled
-communication.
-
-<ThemedImage
-  alt="Docusaurus themed image"
-  sources={{
-    light: useBaseUrl("/img/scenario-3_light.svg"),
-    dark: useBaseUrl("/img/scenario-3_dark.svg"),
-  }}
-/>
-
-## SSH Tunnel Setup for interLink
-
-This guide explains how to configure SSH tunneling between Virtual Kubelet and interLink API server using the built-in `ssh-tunnel` command. SSH tunneling enables secure communication in scenarios where direct network connectivity is not available or desired.
-
-### Overview
-
-The SSH tunnel functionality allows you to:
-
-- Connect Virtual Kubelet to a remote interLink API server through an SSH tunnel
-- Secure communication over untrusted networks
-- Bypass network restrictions and firewalls
-- Enable the **tunneled deployment pattern** where the API server runs locally and the plugin runs remotely
-
-### Architecture
-
-In a tunneled deployment:
-
-1. **Virtual Kubelet** runs in your local Kubernetes cluster
-2. **interLink API server** runs locally (same network as Virtual Kubelet)  
-3. **SSH tunnel** forwards traffic from local Unix socket to remote TCP port
-4. **Plugin** runs on the remote compute resource (HPC cluster, cloud, etc.)
-
-```
-[Virtual Kubelet] -> [interLink API] -> [Unix Socket] -> [SSH Tunnel] -> [Remote Plugin]
-     (local)            (local)          (local)        (ssh bridge)      (remote)
-```
-
-### Prerequisites
-
-Before setting up SSH tunneling, ensure you have:
-
-1. **SSH access** to the remote system where the plugin runs
-2. **SSH key pair** for authentication
-3. **Network connectivity** from local system to remote SSH server
-4. **interLink binary** built with ssh-tunnel command (`make ssh-tunnel`)
-
-#### SSH Key Setup
-
-Generate an SSH key pair if you don't have one:
-
-```bash
-# Generate SSH key pair
-ssh-keygen -t rsa -b 4096 -f ~/.ssh/interlink_rsa
-
-# Copy public key to remote server
-ssh-copy-id -i ~/.ssh/interlink_rsa.pub user@remote-server
-
-# Test SSH connection
-ssh -i ~/.ssh/interlink_rsa user@remote-server
-```
-
-#### Optional: Host Key Verification
-
-For enhanced security, extract the remote server's host key:
-
-```bash
-# Extract host public key from remote server
-ssh-keyscan -t rsa remote-server > ~/.ssh/interlink_host_key
-
-# Or get it from known_hosts
-ssh-keygen -F remote-server -f ~/.ssh/known_hosts | grep -o 'ssh-rsa.*' > ~/.ssh/interlink_host_key
-```
-
-### Configuration
-
-#### Step 1: Configure interLink API Server
-
-Configure the interLink API server to listen on a Unix socket instead of a TCP port:
-
-```yaml title="InterLinkConfig.yaml"
-# Use Unix socket for local communication
-InterlinkAddress: "unix:///tmp/interlink.sock"
-InterlinkPort: ""  # Not used for Unix sockets
-
-# Remote plugin configuration
-SidecarURL: "http://remote-plugin"
-SidecarPort: "4000"
-
-VerboseLogging: true
-ErrorsOnlyLogging: false
-DataRootFolder: "/tmp/interlink"
-```
-
-#### Step 2: Configure Virtual Kubelet
-
-Configure Virtual Kubelet to connect to the Unix socket:
-
-```yaml title="VirtualKubeletConfig.yaml"
-# Connect to Unix socket
-InterlinkURL: "unix:///tmp/interlink.sock"
-InterlinkPort: ""  # Not used for Unix sockets
-
-VerboseLogging: true
-ErrorsOnlyLogging: false
-
-# Node configuration
-NodeName: "my-interlink-node"
-NodeLabels:
-  "interlink.cern.ch/provider": "remote-hpc"
-```
-
-#### Step 3: Start SSH Tunnel
-
-Use the built-in `ssh-tunnel` command to establish the tunnel:
-
-##### Basic Usage
-
-```bash
-# Start SSH tunnel
-./bin/ssh-tunnel \
-  -addr "remote-server:22" \
-  -user "username" \
-  -keyfile "~/.ssh/interlink_rsa" \
-  -lsock "/tmp/interlink.sock" \
-  -rport "4000"
-```
-
-##### With Host Key Verification
-
-```bash
-# Start SSH tunnel with host key verification
-./bin/ssh-tunnel \
-  -addr "remote-server:22" \
-  -user "username" \
-  -keyfile "~/.ssh/interlink_rsa" \
-  -lsock "/tmp/interlink.sock" \
-  -rport "4000" \
-  -hostkeyfile "~/.ssh/interlink_host_key"
-```
-
-##### Command Line Options
-
-| Option | Description | Required |
-|--------|-------------|----------|
-| `-addr` | SSH server address as `hostname:port` | Yes |
-| `-user` | Username for SSH authentication | Yes |
-| `-keyfile` | Path to private key file | Yes |
-| `-lsock` | Path to local Unix socket | Yes |
-| `-rport` | Remote port where plugin listens | Yes |
-| `-hostkeyfile` | Path to host public key for verification | No |
-
-### Complete Deployment Example
-
-#### Step 1: Prepare Remote Environment
-
-On the remote server, start your interLink plugin:
-
-```bash
-# Example: Start SLURM plugin on remote HPC system
-cd /path/to/plugin
-python3 slurm_plugin.py --port 4000
-```
-
-#### Step 2: Start Local Components
-
-Start components in this order:
-
-```bash
-# 1. Start SSH tunnel (runs in foreground)
-./bin/ssh-tunnel \
-  -addr "hpc-cluster.example.com:22" \
-  -user "hpc-user" \
-  -keyfile "~/.ssh/interlink_rsa" \
-  -lsock "/tmp/interlink.sock" \
-  -rport "4000" \
-  -hostkeyfile "~/.ssh/interlink_host_key"
-```
-
-In separate terminals:
-
-```bash
-# 2. Start interLink API server
-export INTERLINKCONFIGPATH=/path/to/InterLinkConfig.yaml
-./bin/interlink
-
-# 3. Start Virtual Kubelet
-export KUBECONFIG=~/.kube/config
-./bin/virtual-kubelet \
-  --provider interlink \
-  --nodename interlink-node \
-  --config /path/to/VirtualKubeletConfig.yaml
-```
-
-#### Step 3: Verify Connection
-
-Test the complete setup:
-
-```bash
-# Check if node appears in Kubernetes
-kubectl get nodes
-
-# Deploy a test pod
-kubectl apply -f - <<EOF
-apiVersion: v1
-kind: Pod
-metadata:
-  name: test-tunnel
-spec:
-  nodeSelector:
-    kubernetes.io/hostname: interlink-node
-  tolerations:
-    - key: virtual-node.interlink/no-schedule
-      operator: Exists
-  containers:
-  - name: test
-    image: busybox
-    command: ["sleep", "3600"]
-EOF
-
-# Check pod status
-kubectl get pod test-tunnel -o wide
-
-# Test logging functionality (requires CSR approval)
-kubectl logs test-tunnel
-```
-
-:::warning CSR Manual Approval Required
-
-⚠️ **IMPORTANT**: If `kubectl logs` fails, you need to manually approve Certificate Signing Requests (CSRs). Without CSR approval, **log access will not work**.
-
-**Required steps:**
-
-1. **Check for pending CSRs**:
-   ```bash
-   kubectl get csr
-   ```
-
-2. **Approve the CSR** (replace `csr-xxxxx` with the actual CSR name):
-   ```bash
-   kubectl certificate approve csr-xxxxx
-   ```
-
-3. **Verify logs are now accessible**:
-   ```bash
-   kubectl logs test-tunnel
-   ```
-
-:::
-
-## Systemd Service Configuration
-
-For production deployments, it's recommended to manage all tunneled components using systemd services. This provides automatic startup, restart on failure, and proper logging.
-
-### Create System User
-
-First, create a dedicated system user for running interLink services:
-
-```bash
-sudo useradd --system --create-home --home-dir /opt/interlink --shell /bin/bash interlink
-sudo mkdir -p /opt/interlink/{bin,config,logs,.ssh}
-sudo chown -R interlink:interlink /opt/interlink
-```
-
-### Copy Binaries and Configuration
-
-Move your interLink components to the system directories:
-
-```bash
-# Copy binaries
-sudo cp ./bin/ssh-tunnel /opt/interlink/bin/
-sudo cp ./bin/interlink /opt/interlink/bin/
-sudo cp ./bin/virtual-kubelet /opt/interlink/bin/
-sudo chmod +x /opt/interlink/bin/*
-
-# Copy configuration files
-sudo cp InterLinkConfig.yaml /opt/interlink/config/
-sudo cp VirtualKubeletConfig.yaml /opt/interlink/config/
-
-# Copy SSH keys
-sudo cp ~/.ssh/interlink_rsa /opt/interlink/.ssh/id_rsa
-sudo cp ~/.ssh/interlink_rsa.pub /opt/interlink/.ssh/id_rsa.pub
-sudo cp ~/.ssh/interlink_host_key /opt/interlink/.ssh/host_key
-
-# Set ownership and permissions
-sudo chown -R interlink:interlink /opt/interlink
-sudo chmod 600 /opt/interlink/.ssh/id_rsa
-sudo chmod 644 /opt/interlink/.ssh/id_rsa.pub
-sudo chmod 644 /opt/interlink/.ssh/host_key
-```
-
-### SSH Tunnel Service
-
-Create the SSH tunnel systemd service:
-
-```ini title="/etc/systemd/system/interlink-tunnel.service"
-[Unit]
-Description=interLink SSH Tunnel
-After=network.target
-Wants=network.target
-
-[Service]
-Type=simple
-User=interlink
-Group=interlink
-WorkingDirectory=/opt/interlink
-ExecStart=/opt/interlink/bin/ssh-tunnel \
-  -addr "remote-server:22" \
-  -user "interlink" \
-  -keyfile "/opt/interlink/.ssh/id_rsa" \
-  -lsock "/tmp/interlink.sock" \
-  -rport "4000" \
-  -hostkeyfile "/opt/interlink/.ssh/host_key"
-
-Restart=always
-RestartSec=10
-StandardOutput=append:/opt/interlink/logs/ssh-tunnel.log
-StandardError=append:/opt/interlink/logs/ssh-tunnel.log
-
-# Security settings
-NoNewPrivileges=true
-ProtectSystem=strict
-ProtectHome=true
-ReadWritePaths=/opt/interlink/logs /tmp
-PrivateTmp=true
-
-[Install]
-WantedBy=multi-user.target
-```
-
-### interLink API Server Service
-
-Create the interLink API server systemd service:
-
-```ini title="/etc/systemd/system/interlink-api.service"
-[Unit]
-Description=interLink API Server (Tunneled)
-After=network.target interlink-tunnel.service
-Wants=network.target
-Requires=interlink-tunnel.service
-
-[Service]
-Type=simple
-User=interlink
-Group=interlink
-WorkingDirectory=/opt/interlink
-Environment=INTERLINKCONFIGPATH=/opt/interlink/config/InterLinkConfig.yaml
-ExecStart=/opt/interlink/bin/interlink
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=always
-RestartSec=10
-StandardOutput=append:/opt/interlink/logs/interlink-api.log
-StandardError=append:/opt/interlink/logs/interlink-api.log
-
-# Security settings
-NoNewPrivileges=true
-ProtectSystem=strict
-ProtectHome=true
-ReadWritePaths=/opt/interlink/logs /tmp
-PrivateTmp=true
-
-[Install]
-WantedBy=multi-user.target
-```
-
-### Virtual Kubelet Service
-
-Create the Virtual Kubelet systemd service:
-
-```ini title="/etc/systemd/system/interlink-virtual-kubelet.service"
-[Unit]
-Description=interLink Virtual Kubelet (Tunneled)
-After=network.target interlink-api.service
-Wants=network.target
-Requires=interlink-api.service
-
-[Service]
-Type=simple
-User=interlink
-Group=interlink
-WorkingDirectory=/opt/interlink
-Environment=KUBECONFIG=/opt/interlink/.kube/config
-ExecStart=/opt/interlink/bin/virtual-kubelet \
-  --provider interlink \
-  --nodename interlink-node \
-  --config /opt/interlink/config/VirtualKubeletConfig.yaml
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=always
-RestartSec=10
-StandardOutput=append:/opt/interlink/logs/virtual-kubelet.log
-StandardError=append:/opt/interlink/logs/virtual-kubelet.log
-
-# Security settings
-NoNewPrivileges=true
-ProtectSystem=strict
-ProtectHome=true
-ReadWritePaths=/opt/interlink/logs /tmp
-PrivateTmp=true
-
-[Install]
-WantedBy=multi-user.target
-```
-
-### Remote Plugin Service (for Remote Server)
-
-For the remote server where the plugin runs, create a systemd service:
-
-```ini title="/etc/systemd/system/interlink-remote-plugin.service"
-[Unit]
-Description=interLink Remote Plugin
-After=network.target
-Wants=network.target
-
-[Service]
-Type=simple
-User=interlink
-Group=interlink
-WorkingDirectory=/opt/interlink-plugin
-Environment=PLUGIN_CONFIG=/opt/interlink-plugin/config/plugin-config.yaml
-ExecStart=/opt/interlink-plugin/bin/plugin
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=always
-RestartSec=10
-StandardOutput=append:/var/log/interlink/plugin.log
-StandardError=append:/var/log/interlink/plugin.log
-
-# Security settings
-NoNewPrivileges=true
-ProtectSystem=strict
-ProtectHome=true
-ReadWritePaths=/var/log/interlink /opt/interlink-plugin/jobs /tmp
-PrivateTmp=true
-
-[Install]
-WantedBy=multi-user.target
-```
-
-### Kubernetes Configuration
-
-Set up the kubeconfig for the interLink user:
-
-```bash
-# Copy kubeconfig for interLink user
-sudo mkdir -p /opt/interlink/.kube
-sudo cp ~/.kube/config /opt/interlink/.kube/config
-sudo chown -R interlink:interlink /opt/interlink/.kube
-sudo chmod 600 /opt/interlink/.kube/config
-```
-
-### Log Rotation Configuration
-
-Create log rotation configuration:
-
-```bash title="/etc/logrotate.d/interlink-tunneled"
-/opt/interlink/logs/*.log {
-    daily
-    rotate 30
-    compress
-    delaycompress
-    missingok
-    notifempty
-    postrotate
-        systemctl reload interlink-tunnel interlink-api interlink-virtual-kubelet 2>/dev/null || true
-    endscript
-}
-
-/var/log/interlink/*.log {
-    daily
-    rotate 30
-    compress
-    delaycompress
-    missingok
-    notifempty
-    postrotate
-        systemctl reload interlink-remote-plugin 2>/dev/null || true
-    endscript
-}
-```
-
-### Service Management Commands
-
-Enable and start all services in the correct order:
-
-```bash
-# Local services (where Virtual Kubelet runs)
-sudo systemctl daemon-reload
-sudo systemctl enable interlink-tunnel interlink-api interlink-virtual-kubelet
-
-# Start services in dependency order
-sudo systemctl start interlink-tunnel
-sudo systemctl start interlink-api
-sudo systemctl start interlink-virtual-kubelet
-
-# Remote services (on the plugin server)
-sudo systemctl daemon-reload
-sudo systemctl enable interlink-remote-plugin
-sudo systemctl start interlink-remote-plugin
-
-# Check service status
-sudo systemctl status interlink-tunnel
-sudo systemctl status interlink-api
-sudo systemctl status interlink-virtual-kubelet
-```
-
-### Service Operations
-
-Common systemd operations for managing tunneled interLink services:
-
-```bash
-# View service logs
-sudo journalctl -u interlink-tunnel -f
-sudo journalctl -u interlink-api -f
-sudo journalctl -u interlink-virtual-kubelet -f
-
-# Restart tunnel (will cascade to dependent services)
-sudo systemctl restart interlink-tunnel
-
-# Stop all local interLink services
-sudo systemctl stop interlink-virtual-kubelet interlink-api interlink-tunnel
-
-# Start all local interLink services
-sudo systemctl start interlink-tunnel interlink-api interlink-virtual-kubelet
-
-# Check service dependencies
-sudo systemctl list-dependencies interlink-virtual-kubelet
-```
-
-### Monitoring and Health Checks
-
-Create a comprehensive health check script for tunneled deployment:
-
-```bash title="/opt/interlink/bin/tunneled-health-check.sh"
-#!/bin/bash
-
-# Health check script for tunneled interLink deployment
-LOG_FILE="/opt/interlink/logs/health-check.log"
-SOCKET_PATH="/tmp/interlink.sock"
-REMOTE_HOST="remote-server"
-REMOTE_PORT="4000"
-
-echo "$(date): Starting tunneled deployment health check" >> "$LOG_FILE"
-
-# Check SSH tunnel connectivity
-if ! pgrep -f "ssh-tunnel" > /dev/null; then
-    echo "$(date): ERROR - SSH tunnel process not running" >> "$LOG_FILE"
-    exit 1
-fi
-
-# Check if Unix socket exists and is responding
-if [ -S "$SOCKET_PATH" ]; then
-    response=$(curl -s --unix-socket "$SOCKET_PATH" http://unix/pinglink 2>/dev/null)
-    if [ $? -eq 0 ]; then
-        echo "$(date): Local API health check passed - $response" >> "$LOG_FILE"
-    else
-        echo "$(date): ERROR - Local API not responding via socket" >> "$LOG_FILE"
-        exit 1
-    fi
-else
-    echo "$(date): ERROR - Unix socket not found at $SOCKET_PATH" >> "$LOG_FILE"
-    exit 1
-fi
-
-# Check Virtual Kubelet node status
-if kubectl get node interlink-node --no-headers 2>/dev/null | grep -q Ready; then
-    echo "$(date): Virtual Kubelet node is Ready" >> "$LOG_FILE"
-else
-    echo "$(date): WARNING - Virtual Kubelet node not Ready" >> "$LOG_FILE"
-fi
-
-# Test remote connectivity through tunnel
-if nc -z -w5 127.0.0.1 4000 2>/dev/null; then
-    echo "$(date): Remote plugin connectivity through tunnel - OK" >> "$LOG_FILE"
-else
-    echo "$(date): WARNING - Cannot reach remote plugin through tunnel" >> "$LOG_FILE"
-fi
-
-echo "$(date): Tunneled deployment health check completed" >> "$LOG_FILE"
-exit 0
-```
-
-```bash
-# Make executable
-sudo chmod +x /opt/interlink/bin/tunneled-health-check.sh
-sudo chown interlink:interlink /opt/interlink/bin/tunneled-health-check.sh
-```
-
-Create systemd timer for health checks:
-
-```ini title="/etc/systemd/system/interlink-tunneled-health-check.service"
-[Unit]
-Description=interLink Tunneled Health Check
-After=interlink-virtual-kubelet.service
-Requires=interlink-virtual-kubelet.service
-
-[Service]
-Type=oneshot
-User=interlink
-Group=interlink
-ExecStart=/opt/interlink/bin/tunneled-health-check.sh
-```
-
-```ini title="/etc/systemd/system/interlink-tunneled-health-check.timer"
-[Unit]
-Description=Run interLink Tunneled Health Check every 5 minutes
-Requires=interlink-tunneled-health-check.service
-
-[Timer]
-OnCalendar=*:0/5
-Persistent=true
-
-[Install]
-WantedBy=timers.target
-```
-
-Enable the health check timer:
-
-```bash
-sudo systemctl daemon-reload
-sudo systemctl enable interlink-tunneled-health-check.timer
-sudo systemctl start interlink-tunneled-health-check.timer
-```
-
-### Troubleshooting Tunneled Deployment
-
-#### SSH Tunnel Issues
-
-```bash
-# Check SSH tunnel process
-ps aux | grep ssh-tunnel
-
-# Test SSH connection manually
-sudo -u interlink ssh -i /opt/interlink/.ssh/id_rsa interlink@remote-server
-
-# Check SSH tunnel logs
-sudo journalctl -u interlink-tunnel --since "1 hour ago"
-
-# Test local socket
-echo "test" | nc -U /tmp/interlink.sock
-```
-
-#### Virtual Kubelet Issues
-
-```bash
-# Check Virtual Kubelet logs
-sudo journalctl -u interlink-virtual-kubelet -f
-
-# Verify kubeconfig access
-sudo -u interlink kubectl get nodes
-
-# Check node status
-kubectl describe node interlink-node
-```
-
-#### Remote Plugin Issues
-
-```bash
-# On remote server, check plugin status
-sudo systemctl status interlink-remote-plugin
-
-# Check if plugin port is listening
-netstat -tlnp | grep :4000
-
-# Test plugin connectivity from remote server
-curl -X GET http://localhost:4000/status
-```
-
-### Security Considerations for Tunneled Deployment
-
-#### SSH Security
-
-1. **Dedicated SSH keys**: Use separate keys for interLink tunneling
-2. **Key restrictions**: Add restrictions in `authorized_keys`:
-
-```bash
-# On remote server in ~/.ssh/authorized_keys
-command="/usr/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding ssh-rsa AAAAB3... interlink-tunnel-key
-```
-
-3. **SSH configuration**: Secure SSH server configuration:
-
-```bash title="/etc/ssh/sshd_config.d/interlink.conf"
-# Dedicated configuration for interLink tunnel user
-Match User interlink
-    AllowTcpForwarding yes
-    AllowStreamLocalForwarding yes
-    PermitTunnel no
-    X11Forwarding no
-    AllowAgentForwarding no
-    PermitTTY no
-    ForceCommand /bin/false
-```
-
-#### Network Security
-
-```bash
-# Firewall rules for local server
-sudo ufw allow in on lo
-sudo ufw allow out 22/tcp comment "SSH for tunnel"
-
-# Firewall rules for remote server  
-sudo ufw allow from <local-server-ip> to any port 22 comment "SSH tunnel"
-sudo ufw allow 4000/tcp comment "Plugin API"
-```
-
-#### File Permissions
-
-```bash
-# Secure SSH directory
-sudo chmod 700 /opt/interlink/.ssh
-sudo chmod 600 /opt/interlink/.ssh/id_rsa
-sudo chmod 644 /opt/interlink/.ssh/id_rsa.pub /opt/interlink/.ssh/host_key
-
-# Secure configuration files
-sudo chmod 640 /opt/interlink/config/*
-sudo chown root:interlink /opt/interlink/config/*
-```
-
-This comprehensive tunneled deployment setup provides a robust, secure, and manageable solution for connecting Kubernetes clusters to remote compute resources through SSH tunneling.
-
-:::note
-
-For additional case studies and advanced configurations, reach out to the interLink community through the Slack channel.
-
-:::
diff --git a/docs/docs/guides/08-systemd-deployment.mdx b/docs/docs/guides/08-systemd-deployment.mdx
index 0c82eda6..eca2a9ee 100644
--- a/docs/docs/guides/08-systemd-deployment.mdx
+++ b/docs/docs/guides/08-systemd-deployment.mdx
@@ -91,6 +91,69 @@ PrivateTmp=true
 WantedBy=multi-user.target
 ```
 
+### SSH Tunnel Service
+
+Only needed for the [in-cluster deployment with a remote plugin](../cookbook/2-incluster.mdx#plugin-on-the-remote-system),
+where the API server reaches its plugin through an SSH tunnel terminating on a
+local Unix socket. Skip this if the plugin runs locally.
+
+```ini title="/etc/systemd/system/interlink-tunnel.service"
+[Unit]
+Description=interLink SSH Tunnel
+After=network.target
+Wants=network.target
+
+[Service]
+Type=simple
+User=interlink
+Group=interlink
+WorkingDirectory=/opt/interlink
+ExecStart=/opt/interlink/bin/ssh-tunnel \
+  -addr "remote-server:22" \
+  -user "interlink" \
+  -keyfile "/opt/interlink/.ssh/id_rsa" \
+  -lsock "/tmp/interlink.sock" \
+  -rport "4000" \
+  -hostkeyfile "/opt/interlink/.ssh/host_key"
+
+Restart=always
+RestartSec=10
+StandardOutput=append:/opt/interlink/logs/ssh-tunnel.log
+StandardError=append:/opt/interlink/logs/ssh-tunnel.log
+
+# Security settings
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+ReadWritePaths=/opt/interlink/logs /tmp
+PrivateTmp=true
+
+[Install]
+WantedBy=multi-user.target
+```
+
+The SSH key material lives under `/opt/interlink/.ssh` and must be readable only
+by the service user:
+
+```bash
+sudo mkdir -p /opt/interlink/.ssh
+sudo cp ~/.ssh/interlink_rsa /opt/interlink/.ssh/id_rsa
+sudo cp ~/.ssh/interlink_host_key /opt/interlink/.ssh/host_key
+sudo chown -R interlink:interlink /opt/interlink/.ssh
+sudo chmod 700 /opt/interlink/.ssh
+sudo chmod 600 /opt/interlink/.ssh/id_rsa
+sudo chmod 644 /opt/interlink/.ssh/host_key
+```
+
+When the tunnel is in use, make the API server depend on it so systemd starts
+them in the right order and restarts the API server if the tunnel is bounced:
+
+```ini title="/etc/systemd/system/interlink-api.service.d/tunnel.conf"
+[Unit]
+After=interlink-tunnel.service
+Requires=interlink-tunnel.service
+```
+
 ### InterLink API Server Service
 
 Create the InterLink API server systemd service:
diff --git a/docs/docs/intro.mdx b/docs/docs/intro.mdx
index 9b895c8c..e62a87d8 100644
--- a/docs/docs/intro.mdx
+++ b/docs/docs/intro.mdx
@@ -38,12 +38,12 @@ interface and [interLink API plugins spec](./guides/03-api-reference.mdx).
 Plugins will define how the containers will run on the remote system (see
 [Target providers](#providers)).
 
-InterLink API and the plugin can be arranged in three different ways across the
+InterLink API and the plugin can be arranged in two different ways across the
 local cluster and the remote part:
 
 - both deployed remote (**[Edge-node](#edge-node)**)
-- both deployed local (**[In-cluster](#in-cluster)**)
-- API local, plugin remote (**[Tunneled](#tunneled)**)
+- API deployed in the cluster (**[In-cluster](#in-cluster)**), with the plugin
+  either beside it or on the remote system, reached over an SSH tunnel
 
 ```
 +---------------------------+  +----------------------------+
@@ -65,6 +65,9 @@ local cluster and the remote part:
 +---------------------------+  +----------------------------+
 ```
 
+The first two rows are both *in-cluster* deployments — the API server sits in the
+cluster either way, and only the plugin moves. The third is the *edge-node* one.
+
 For more information visit the [architecture page](arch)
 
 ## Target applications
@@ -151,9 +154,13 @@ Get started with edge-node deployment [here](./cookbook/1-edge.mdx)
 ### In-cluster
 
 This scenario involves deploying a Virtual Kubelet along with the interLink API
-server and the plugin to interact with a remote API. This setup allows
-Kubernetes pods to be executed on remote resources while all other components
-sits inside the Kubernetes cluster.
+server inside the Kubernetes cluster, so nothing has to be installed on an edge
+node and no interLink component is exposed to the internet.
+
+The plugin either runs beside them in the cluster, or on the remote system — on a
+login node, for instance — with the API server reaching it through an SSH tunnel.
+That second variant suits multi-user hosts, and sites that allow outbound SSH but
+will not let you expose a port.
 
 <ThemedImage
   alt="Docusaurus themed image"
@@ -169,24 +176,3 @@ Get started with in-cluster deployment [here](./cookbook/2-incluster.mdx)
 
 :::
 
-### Tunneled
-
-This deployment involves the Virtual Kubelet connecting to a remote interLink
-API server and its plugin through a secure tunnel. This setup ensures secure
-communication between the Kubernetes cluster and the remote resources, making it
-suitable for environments with strict security requirements or to host services
-on a multi user host like a login node.
-
-<ThemedImage
-  alt="Docusaurus themed image"
-  sources={{
-    light: useBaseUrl("/img/scenario-3_light.svg"),
-    dark: useBaseUrl("/img/scenario-3_dark.svg"),
-  }}
-/>
-
-:::note
-
-Get started with tunneled deployment [here](./cookbook/3-tunneled.mdx)
-
-:::

From 55380d53566cebe9ace32e47285f75397665cf34 Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 31 Jul 2026 11:38:37 +0200
Subject: [PATCH 3/5] docs: state what the login node has to allow for the SSH
 shadow
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Validating the SSH shadow against a real site turned up the one prerequisite the
guide never named: sshd has to permit TCP forwarding for the account. Sites that
set "AllowTcpForwarding no" globally and re-enable it per group are common, and
the failure is silent until traffic arrives — the shadow starts, reports Ready,
and then resets every connection with only "administratively prohibited" in its
log.

Add a prerequisites section with a check to run before deploying, the two other
things the login node has to provide (a route to the compute nodes on the exposed
ports, and inbound SSH from the cluster), and a troubleshooting entry keyed on the
exact error string.

Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .../guides/14-ssh-tunnel-configuration.mdx    | 51 +++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index 73694126..37f644d4 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -33,6 +33,52 @@ storage or message buses. Combining the two is rejected at startup and tracked i
 
 ---
 
+## Before you start: what the login node has to allow
+
+The shadow is an ordinary SSH client, so everything it needs is decided by the
+login node's `sshd_config`.
+
+1. **`AllowTcpForwarding yes`** for the account the shadow logs in as. This is the
+   one that catches people out: plenty of HPC sites set `AllowTcpForwarding no`
+   globally and re-enable it only for a subset of users, often only those who
+   authenticate with MFA. Check it before anything else:
+
+   ```bash
+   # from your own machine, with the same key/principal the shadow will use
+   ssh -N -L 19999:<login node>:22 <user>@<login node> &
+   nc -z 127.0.0.1 19999 && head -c 40 < /dev/tcp/127.0.0.1/19999
+   ```
+
+   If the site refuses, the shadow still starts and still reports Ready — the
+   failure only shows up when traffic arrives, as a connection reset at the client
+   and this line in `kubectl logs <shadow pod> -c ssh-forward`:
+
+   ```
+   channel 1: open failed: administratively prohibited: open failed
+   ```
+
+   There is no way around it from the cluster side; ask the site to allow TCP
+   forwarding for the account, optionally narrowed with `PermitOpen`.
+
+2. **A route from the login node to the compute nodes**, on the ports the pod
+   exposes. `ssh -L` resolves and connects to the compute node *from the login
+   node*, using whatever name the plugin reported. Verify with the name the plugin
+   actually reports (`hostname -f` on the compute node for the Slurm plugin), not
+   the short name:
+
+   ```bash
+   ssh <user>@<login node> "curl -sv http://<compute node fqdn>:<port>/"
+   ```
+
+3. **Outbound SSH from the cluster** to the login node's port, from the namespaces
+   shadows are created in. Network policies that restrict pod egress have to allow
+   it.
+
+`ssh -L` carries TCP only. UDP ports on the offloaded pod are skipped, with a
+warning in the virtual kubelet log.
+
+---
+
 ## Which shadow does what
 
 | | Direction | Trigger | Workload must run |
@@ -263,6 +309,11 @@ it. With `ReplicateCredentials: false` the Secret must already exist there.
 nothing is listening on that port. Services often bind `127.0.0.1` by default; bind
 `0.0.0.0` instead — for a notebook, `--ip=0.0.0.0`.
 
+**Connection reset, and `channel N: open failed: administratively prohibited` in the
+`ssh-forward` log.** The login node refuses port forwarding for this account. See
+[what the login node has to allow](#before-you-start-what-the-login-node-has-to-allow);
+this is a site policy and cannot be worked around from the cluster.
+
 **Host key verification failed.** `KnownHostsConfigMap` does not contain the login
 node's current key. Re-run `ssh-keyscan` and update the ConfigMap.
 

From 91b845f61b5b668eea4f5398b834b66ee401b6cf Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 31 Jul 2026 14:15:37 +0200
Subject: [PATCH 4/5] docs: document the exec forward mode

Sites that refuse TCP forwarding now have an option other than "ask the site":
Network.SSH.ForwardMode: exec relays each connection through a command on the
login node. Describe both modes side by side, what each needs from the login
node, and the two things that decide whether exec is usable at a given site --
which netcat is installed, and whether the site tolerates a process per
connection.

Point the prerequisites and troubleshooting entries at it, so the reader who
just hit "administratively prohibited" lands on the alternative rather than a
dead end.

Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .../guides/14-ssh-tunnel-configuration.mdx    | 63 +++++++++++++++++--
 1 file changed, 58 insertions(+), 5 deletions(-)

diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index 37f644d4..6d793b21 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -57,8 +57,9 @@ login node's `sshd_config`.
    channel 1: open failed: administratively prohibited: open failed
    ```
 
-   There is no way around it from the cluster side; ask the site to allow TCP
-   forwarding for the account, optionally narrowed with `PermitOpen`.
+   If the site will not allow it, switch to
+   [`ForwardMode: exec`](#forward-modes), which relays through a command on the
+   login node instead and needs no forwarding privilege.
 
 2. **A route from the login node to the compute nodes**, on the ports the pod
    exposes. `ssh -L` resolves and connects to the compute node *from the login
@@ -132,6 +133,8 @@ in the chart repository for a complete deployment.
 | `Port` | `22` | Login node's SSH port |
 | `Image` | `ghcr.io/interlink-hq/interlink/ssh-tunnel:<version>` | Image the shadow runs. Needs an ssh client, plus `kinit` for Kerberos |
 | `Auth` | `publickey` | `publickey` or `kerberos` |
+| `ForwardMode` | `portforward` | `portforward` (`ssh -L`) or `exec` (relay through a command on the login node) |
+| `ExecConnectCommand` | `nc` | Command run on the login node in `exec` mode, invoked as `<command> <node> <port>` |
 | `KeySecret` | — | Secret holding the private key (`publickey`) |
 | `KeySecretKey` | `id_ed25519` | Key inside `KeySecret` |
 | `KeytabSecret` | — | Secret holding the keytab (`kerberos`) |
@@ -148,6 +151,57 @@ Misconfiguration fails at **startup**, not on the first offloaded pod, so a miss
 
 ---
 
+## Forward modes
+
+The default, `portforward`, is one `ssh -L` per exposed port. It is the cheapest
+option and depends on nothing but the ssh client, but it requires
+[`AllowTcpForwarding yes`](#before-you-start-what-the-login-node-has-to-allow) on
+the login node.
+
+`exec` is for sites that do not grant it. It gives each exposed port a local
+listener in the shadow and relays every accepted connection through a command run
+on the login node — by default `nc <compute node> <port>`. Nothing asks sshd to
+forward anything, so the only privileges involved are the ones the shadow already
+uses to log in.
+
+```yaml
+Network:
+  ShadowMode: "ssh"
+  SSH:
+    LoginHost: "login.hpc.example.org"
+    User: "alice"
+    KeySecret: "hpc-ssh-key"
+    ForwardMode: "exec"
+    ExecConnectCommand: "nc"   # ncat, or "nc -q 1" on a netcat that does not half-close
+```
+
+The two are interchangeable above the socket: the Service still fronts the same
+container port, and the offloaded workload still only binds `0.0.0.0`. Nothing on
+either side can tell them apart.
+
+| | `portforward` | `exec` |
+| --- | --- | --- |
+| Login node must allow | TCP forwarding | running a command, and have a netcat |
+| Connections | one long-lived ssh | one ssh channel per TCP connection |
+| Cost per connection | none | one multiplexed channel open |
+| Failure surfaces | at connect time | at connect time |
+
+The relays share a single SSH connection, opened as a control master before the
+listeners start. Without it every connection would pay a full handshake — a
+notebook UI opens dozens — and the login node would see a session per connection.
+Because they share one TCP connection they also share its head-of-line blocking,
+which is fine for a UI and less so for bulk transfer.
+
+Two things to check when picking `exec`:
+
+- **Which netcat.** `nc` varies. `netcat-traditional` does not propagate a
+  half-close, so a client that closes its write side can leave the connection
+  hanging; `nc -q 1` or `ncat` avoids it. Set `ExecConnectCommand` accordingly.
+- **Process limits.** Even multiplexed, the login node runs a netcat per
+  connection. Sites with per-user process caps will notice a busy shadow.
+
+---
+
 ## Authentication
 
 ### Public key
@@ -310,9 +364,8 @@ nothing is listening on that port. Services often bind `127.0.0.1` by default; b
 `0.0.0.0` instead — for a notebook, `--ip=0.0.0.0`.
 
 **Connection reset, and `channel N: open failed: administratively prohibited` in the
-`ssh-forward` log.** The login node refuses port forwarding for this account. See
-[what the login node has to allow](#before-you-start-what-the-login-node-has-to-allow);
-this is a site policy and cannot be worked around from the cluster.
+`ssh-forward` log.** The login node refuses port forwarding for this account. Either
+have the site allow it, or switch to [`ForwardMode: exec`](#forward-modes).
 
 **Host key verification failed.** `KnownHostsConfigMap` does not contain the login
 node's current key. Re-run `ssh-keyscan` and update the ConfigMap.

From bc108e001816dd60a3e0565ce983b3d762f606a7 Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 31 Jul 2026 17:01:01 +0200
Subject: [PATCH 5/5] docs: cover the behaviours added while validating the SSH
 shadow

Three things the guide described as they were before deployment testing changed
them, each of which a reader can now hit and find nothing about.

The compute node section stopped at "the tunnel starts". It now also covers a
requeued job moving the tunnel in place, which is a log line an operator will see
and should not read as a fault, and states what a plugin is allowed to report:
interLink refuses anything that is not a hostname or an IP, because the value ends
up as an ssh argument, so a plugin returning something else fails in a way worth
naming.

Credential replication gained a marker and a refusal. A name collision in a
shadow namespace now fails pod creation instead of destroying whatever was there,
which is a message someone will search for.

Two matching troubleshooting entries, both keyed on the exact log line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
---
 .../guides/14-ssh-tunnel-configuration.mdx    | 51 +++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index 6d793b21..2b1b6f53 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -278,11 +278,38 @@ While the job is queued the shadow sits in `Init:0/1`. That is the expected stat
 not a failure — it reads as "waiting for the allocation" rather than a crash-looping
 tunnel. After `NodeWaitTimeout` the init container fails with an explicit message.
 
+If the job is later requeued onto a different host, the plugin reports the new one,
+interLink republishes it, and the shadow rebuilds its forwarders **in place**:
+
+```
+compute node changed from node07.hpc.example.org to node11.hpc.example.org, rebuilding the tunnel
+```
+
+The container is not restarted, so the shadow keeps its pod IP — the one Kubernetes
+has already been told is the offloaded pod's IP.
+
 :::warning
 This requires a plugin that reports `PodStatus.NodeName`. A plugin that does not
 will leave the shadow waiting until the timeout expires.
 :::
 
+### What a plugin may report
+
+The reported name ends up as an argument to `ssh`, so interLink accepts only an
+RFC 1123 hostname or an IP literal and refuses anything else rather than trying to
+escape it:
+
+```
+Refusing to publish compute node "node07 -oProxyCommand=..." for ns/pod:
+not a hostname or IP address. The shadow passes this to ssh, so it is rejected
+rather than escaped.
+```
+
+A name carrying whitespace would otherwise split into an extra ssh option, and one
+carrying a quote or `$(` would break out of the relay command in `exec` mode. Plugins
+should report what the node calls itself — `hostname -f` for the Slurm plugin — and
+nothing else.
+
 ---
 
 ## Credential replication
@@ -294,6 +321,20 @@ shadow lands in a per-user namespace, where the SSH credential does not exist.
 By default interLink copies the credential Secret, and any `Krb5ConfigMap` /
 `KnownHostsConfigMap`, from its own namespace into the shadow's namespace.
 
+Copies are marked with `interlink.eu/replicated-from`, and only objects carrying
+that annotation are overwritten on a later pass. If an object of the same name is
+already there and holds something else, pod creation fails rather than replacing it:
+
+```
+refusing to overwrite secret alice/hpc-ssh-key: it already exists, holds different
+content and was not created by interLink
+```
+
+That matters with `shadow-same-ns`, where the target is somebody's own namespace and
+a name collision would otherwise destroy their Secret. An unmarked copy whose content
+already matches is adopted instead of refused, so upgrading from a version that
+predates the marker does not break existing deployments.
+
 :::warning
 Replication makes the credential readable by anyone who can read Secrets in those
 namespaces. If that is not acceptable, set `ReplicateCredentials: false` and
@@ -370,6 +411,16 @@ have the site allow it, or switch to [`ForwardMode: exec`](#forward-modes).
 **Host key verification failed.** `KnownHostsConfigMap` does not contain the login
 node's current key. Re-run `ssh-keyscan` and update the ConfigMap.
 
+**`Refusing to publish compute node ...` in the virtual kubelet log, and the shadow
+stays in `Init:0/1`.** The plugin reported something that is not a hostname or an IP
+address, so it was not passed to ssh. See
+[what a plugin may report](#what-a-plugin-may-report); the plugin needs fixing, and
+the value it produced is quoted in the message.
+
+**Pod creation fails with `refusing to overwrite secret ...`.** The shadow's namespace
+already contains a different object under the credential's name. Rename the
+credential, or set `ReplicateCredentials: false` and provision it yourself.
+
 **An nginx wstunnel Deployment appears instead of the SSH shadow.** A custom template
 file is overriding the built-in one. The virtual kubelet prefers
 `Network.WstunnelTemplatePath` over its own templates; the Helm chart stops mounting