diff --git a/docs/docs/cookbook/1-edge.mdx b/docs/docs/cookbook/1-edge.mdx index 23d0739ad..3f15f4401 100644 --- a/docs/docs/cookbook/1-edge.mdx +++ b/docs/docs/cookbook/1-edge.mdx @@ -1,5 +1,5 @@ --- -sidebar_position: 3 +sidebar_position: 1 --- import Tabs from "@theme/Tabs"; diff --git a/docs/docs/cookbook/2-incluster.mdx b/docs/docs/cookbook/2-incluster.mdx index ac3969861..f4bb24b2a 100644 --- a/docs/docs/cookbook/2-incluster.mdx +++ b/docs/docs/cookbook/2-incluster.mdx @@ -1,5 +1,5 @@ --- -sidebar_position: 3 +sidebar_position: 2 --- import Tabs from "@theme/Tabs"; @@ -9,7 +9,9 @@ import useBaseUrl from "@docusaurus/useBaseUrl"; # In-cluster deployment -Deploy interLink in the local K8S cluster. +The interLink API server runs inside your Kubernetes cluster, next to the virtual +kubelet. Nothing has to be installed on an edge node, and no interLink component +is exposed to the internet. -## Install interLink +The remaining choice is where the **plugin** runs, and how the API server reaches +it: + +| | Plugin runs | API to plugin link | Use when | +| --- | --- | --- | --- | +| [In the cluster](#plugin-in-the-cluster) | as a container in the same pod | localhost | the plugin can reach the remote system on its own — a shared filesystem, an SSH shim, a REST API | +| [On the remote system](#plugin-on-the-remote-system) | on the login node or an edge host | SSH tunnel over a Unix socket | the plugin has to run where the batch system is, and you cannot expose a port for it | + +If instead you want the API server *and* the plugin to run on the remote side, see +the [edge node deployment](./1-edge.mdx). + +--- + +## Plugin in the cluster + +Everything runs in one pod: virtual kubelet, interLink API server and plugin. ### Deploy Kubernetes components -The deployment of the Kubernetes components are managed by the official -[HELM chart](https://github.com/interlink-hq/interlink-helm-chart). Depending on -the scenario you selected, there might be additional operations to be done. +The deployment of the Kubernetes components is managed by the official +[HELM chart](https://github.com/interlink-hq/interlink-helm-chart). - Create an helm values file: @@ -127,6 +143,205 @@ You can find a demo pod to test your setup To start debugging in case of problems we suggest starting from the pod containers logs! +### Reaching the batch system from inside the cluster + +The plugin still has to submit jobs somewhere. Which mechanism it uses is a +plugin concern rather than an interLink one, but the common ones are: + +- **Shared filesystem plus SSH shims.** Mount the remote scratch area into the + plugin container, and point the plugin's `SbatchPath` / `SqueuePath` / + `ScancelPath` at small wrappers that `exec ssh user@login /usr/bin/`. The + plugin never has to know it is not running on the login node. +- **A plugin that speaks a remote API** — Kubernetes, a cloud batch service, a + site REST endpoint. + +If neither fits, run the plugin on the remote system instead. + +--- + +## Plugin on the remote system + +Some sites will not let you expose a port for the plugin, but do allow outbound +SSH. In that case the plugin runs on the login node and the API server reaches it +through an SSH tunnel terminating on a local Unix socket. + + + +``` +[Virtual Kubelet] -> [interLink API] -> [Unix socket] -> [SSH tunnel] -> [Plugin] + (local) (local) (local) (ssh bridge) (remote) +``` + +:::info + +This tunnel carries interLink's **control plane** — the API server talking to its +plugin. It is unrelated to the +[SSH shadow](../guides/14-ssh-tunnel-configuration.mdx), which carries traffic +*into* an already-running offloaded pod. A deployment can use either, both, or +neither. + +::: + +### Prerequisites + +1. **SSH access** to the remote system where the plugin runs +2. **SSH key pair** for authentication +3. **Network connectivity** from the local system to the remote SSH server +4. **interLink binary** built with the ssh-tunnel command (`make ssh-tunnel`) + +#### SSH key setup + +```bash +# Generate SSH key pair +ssh-keygen -t rsa -b 4096 -f ~/.ssh/interlink_rsa + +# Copy public key to remote server +ssh-copy-id -i ~/.ssh/interlink_rsa.pub user@remote-server + +# Test SSH connection +ssh -i ~/.ssh/interlink_rsa user@remote-server +``` + +#### Optional: host key verification + +```bash +# Extract host public key from remote server +ssh-keyscan -t rsa remote-server > ~/.ssh/interlink_host_key + +# Or get it from known_hosts +ssh-keygen -F remote-server -f ~/.ssh/known_hosts | grep -o 'ssh-rsa.*' > ~/.ssh/interlink_host_key +``` + +### Step 1: point the API server at a Unix socket + +```yaml title="InterLinkConfig.yaml" +# Use Unix socket for local communication +InterlinkAddress: "unix:///tmp/interlink.sock" +InterlinkPort: "" # Not used for Unix sockets + +# Remote plugin configuration +SidecarURL: "http://remote-plugin" +SidecarPort: "4000" + +VerboseLogging: true +ErrorsOnlyLogging: false +DataRootFolder: "/tmp/interlink" +``` + +### Step 2: point the virtual kubelet at the same socket + +```yaml title="VirtualKubeletConfig.yaml" +InterlinkURL: "unix:///tmp/interlink.sock" +InterlinkPort: "" # Not used for Unix sockets + +VerboseLogging: true +ErrorsOnlyLogging: false + +NodeName: "my-interlink-node" +NodeLabels: + "interlink.cern.ch/provider": "remote-hpc" +``` + +### Step 3: start the tunnel + +```bash +./bin/ssh-tunnel \ + -addr "remote-server:22" \ + -user "username" \ + -keyfile "~/.ssh/interlink_rsa" \ + -lsock "/tmp/interlink.sock" \ + -rport "4000" \ + -hostkeyfile "~/.ssh/interlink_host_key" # optional, but recommended +``` + +| Option | Description | Required | +| --- | --- | --- | +| `-addr` | SSH server address as `hostname:port` | Yes | +| `-user` | Username for SSH authentication | Yes | +| `-keyfile` | Path to private key file | Yes | +| `-lsock` | Path to local Unix socket | Yes | +| `-rport` | Remote port where the plugin listens | Yes | +| `-hostkeyfile` | Path to host public key for verification | No | + +Start the components in dependency order: tunnel, then API server, then virtual +kubelet. + +:::note + +To run these as managed services, see the +[systemd deployment guide](../guides/08-systemd-deployment.mdx), which includes +the SSH tunnel unit and the ordering constraints between the three. + +::: + +### Hardening the tunnel account + +Restrict what the tunnel key is allowed to do on the remote side: + +```bash title="~/.ssh/authorized_keys (remote)" +command="/usr/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding ssh-rsa AAAAB3... interlink-tunnel-key +``` + +```bash title="/etc/ssh/sshd_config.d/interlink.conf" +Match User interlink + AllowTcpForwarding yes + AllowStreamLocalForwarding yes + PermitTunnel no + X11Forwarding no + AllowAgentForwarding no + PermitTTY no + ForceCommand /bin/false +``` + +### Troubleshooting + +```bash +# Is the tunnel process alive? +ps aux | grep ssh-tunnel + +# Does the socket answer? +curl -s --unix-socket /tmp/interlink.sock http://unix/pinglink + +# Is the plugin listening on the remote side? +ssh user@remote-server 'netstat -tlnp | grep :4000' +``` + +--- + +## Verify the setup + +```bash +# Check if node appears in Kubernetes +kubectl get nodes + +# Deploy a test pod +kubectl apply -f - < ``` ::: + +--- + +## Reaching services inside offloaded pods + +Everything above gets *jobs* onto the remote system. Reaching a service that runs +inside an offloaded pod — a notebook, a dashboard — is a separate concern, handled +by a shadow pod that interLink creates for any offloaded pod with exposed ports. + +| | Direction | Requires | Guide | +| --- | --- | --- | --- | +| wstunnel | cluster to pod | outbound internet from the compute node, and a public ingress on the cluster | [Wstunnel](../guides/10-wstunnel-configuration.mdx) | +| SSH | cluster to pod | outbound SSH from the cluster to a login node | [SSH tunnel](../guides/14-ssh-tunnel-configuration.mdx) | +| Full mesh | bidirectional | same as wstunnel, plus an unprivileged network namespace on the compute node | [Mesh network](../guides/13-mesh-network-configuration.mdx) | + +Air-gapped sites — compute nodes with no route out, clusters with no public +ingress — generally want the SSH one. + +:::note + +For additional case studies and advanced configurations, reach out to the +interLink community through the Slack channel. + +::: diff --git a/docs/docs/cookbook/3-tunneled.mdx b/docs/docs/cookbook/3-tunneled.mdx deleted file mode 100644 index a891069cd..000000000 --- a/docs/docs/cookbook/3-tunneled.mdx +++ /dev/null @@ -1,735 +0,0 @@ ---- -sidebar_position: 3 ---- - -import Tabs from "@theme/Tabs"; -import TabItem from "@theme/TabItem"; -import ThemedImage from "@theme/ThemedImage"; -import useBaseUrl from "@docusaurus/useBaseUrl"; - -# Tunneled deployment - -Deploy interLink components in both systems, linked through a tunnelled -communication. - - - -## SSH Tunnel Setup for interLink - -This guide explains how to configure SSH tunneling between Virtual Kubelet and interLink API server using the built-in `ssh-tunnel` command. SSH tunneling enables secure communication in scenarios where direct network connectivity is not available or desired. - -### Overview - -The SSH tunnel functionality allows you to: - -- Connect Virtual Kubelet to a remote interLink API server through an SSH tunnel -- Secure communication over untrusted networks -- Bypass network restrictions and firewalls -- Enable the **tunneled deployment pattern** where the API server runs locally and the plugin runs remotely - -### Architecture - -In a tunneled deployment: - -1. **Virtual Kubelet** runs in your local Kubernetes cluster -2. **interLink API server** runs locally (same network as Virtual Kubelet) -3. **SSH tunnel** forwards traffic from local Unix socket to remote TCP port -4. **Plugin** runs on the remote compute resource (HPC cluster, cloud, etc.) - -``` -[Virtual Kubelet] -> [interLink API] -> [Unix Socket] -> [SSH Tunnel] -> [Remote Plugin] - (local) (local) (local) (ssh bridge) (remote) -``` - -### Prerequisites - -Before setting up SSH tunneling, ensure you have: - -1. **SSH access** to the remote system where the plugin runs -2. **SSH key pair** for authentication -3. **Network connectivity** from local system to remote SSH server -4. **interLink binary** built with ssh-tunnel command (`make ssh-tunnel`) - -#### SSH Key Setup - -Generate an SSH key pair if you don't have one: - -```bash -# Generate SSH key pair -ssh-keygen -t rsa -b 4096 -f ~/.ssh/interlink_rsa - -# Copy public key to remote server -ssh-copy-id -i ~/.ssh/interlink_rsa.pub user@remote-server - -# Test SSH connection -ssh -i ~/.ssh/interlink_rsa user@remote-server -``` - -#### Optional: Host Key Verification - -For enhanced security, extract the remote server's host key: - -```bash -# Extract host public key from remote server -ssh-keyscan -t rsa remote-server > ~/.ssh/interlink_host_key - -# Or get it from known_hosts -ssh-keygen -F remote-server -f ~/.ssh/known_hosts | grep -o 'ssh-rsa.*' > ~/.ssh/interlink_host_key -``` - -### Configuration - -#### Step 1: Configure interLink API Server - -Configure the interLink API server to listen on a Unix socket instead of a TCP port: - -```yaml title="InterLinkConfig.yaml" -# Use Unix socket for local communication -InterlinkAddress: "unix:///tmp/interlink.sock" -InterlinkPort: "" # Not used for Unix sockets - -# Remote plugin configuration -SidecarURL: "http://remote-plugin" -SidecarPort: "4000" - -VerboseLogging: true -ErrorsOnlyLogging: false -DataRootFolder: "/tmp/interlink" -``` - -#### Step 2: Configure Virtual Kubelet - -Configure Virtual Kubelet to connect to the Unix socket: - -```yaml title="VirtualKubeletConfig.yaml" -# Connect to Unix socket -InterlinkURL: "unix:///tmp/interlink.sock" -InterlinkPort: "" # Not used for Unix sockets - -VerboseLogging: true -ErrorsOnlyLogging: false - -# Node configuration -NodeName: "my-interlink-node" -NodeLabels: - "interlink.cern.ch/provider": "remote-hpc" -``` - -#### Step 3: Start SSH Tunnel - -Use the built-in `ssh-tunnel` command to establish the tunnel: - -##### Basic Usage - -```bash -# Start SSH tunnel -./bin/ssh-tunnel \ - -addr "remote-server:22" \ - -user "username" \ - -keyfile "~/.ssh/interlink_rsa" \ - -lsock "/tmp/interlink.sock" \ - -rport "4000" -``` - -##### With Host Key Verification - -```bash -# Start SSH tunnel with host key verification -./bin/ssh-tunnel \ - -addr "remote-server:22" \ - -user "username" \ - -keyfile "~/.ssh/interlink_rsa" \ - -lsock "/tmp/interlink.sock" \ - -rport "4000" \ - -hostkeyfile "~/.ssh/interlink_host_key" -``` - -##### Command Line Options - -| Option | Description | Required | -|--------|-------------|----------| -| `-addr` | SSH server address as `hostname:port` | Yes | -| `-user` | Username for SSH authentication | Yes | -| `-keyfile` | Path to private key file | Yes | -| `-lsock` | Path to local Unix socket | Yes | -| `-rport` | Remote port where plugin listens | Yes | -| `-hostkeyfile` | Path to host public key for verification | No | - -### Complete Deployment Example - -#### Step 1: Prepare Remote Environment - -On the remote server, start your interLink plugin: - -```bash -# Example: Start SLURM plugin on remote HPC system -cd /path/to/plugin -python3 slurm_plugin.py --port 4000 -``` - -#### Step 2: Start Local Components - -Start components in this order: - -```bash -# 1. Start SSH tunnel (runs in foreground) -./bin/ssh-tunnel \ - -addr "hpc-cluster.example.com:22" \ - -user "hpc-user" \ - -keyfile "~/.ssh/interlink_rsa" \ - -lsock "/tmp/interlink.sock" \ - -rport "4000" \ - -hostkeyfile "~/.ssh/interlink_host_key" -``` - -In separate terminals: - -```bash -# 2. Start interLink API server -export INTERLINKCONFIGPATH=/path/to/InterLinkConfig.yaml -./bin/interlink - -# 3. Start Virtual Kubelet -export KUBECONFIG=~/.kube/config -./bin/virtual-kubelet \ - --provider interlink \ - --nodename interlink-node \ - --config /path/to/VirtualKubeletConfig.yaml -``` - -#### Step 3: Verify Connection - -Test the complete setup: - -```bash -# Check if node appears in Kubernetes -kubectl get nodes - -# Deploy a test pod -kubectl apply -f - </dev/null || true - endscript -} - -/var/log/interlink/*.log { - daily - rotate 30 - compress - delaycompress - missingok - notifempty - postrotate - systemctl reload interlink-remote-plugin 2>/dev/null || true - endscript -} -``` - -### Service Management Commands - -Enable and start all services in the correct order: - -```bash -# Local services (where Virtual Kubelet runs) -sudo systemctl daemon-reload -sudo systemctl enable interlink-tunnel interlink-api interlink-virtual-kubelet - -# Start services in dependency order -sudo systemctl start interlink-tunnel -sudo systemctl start interlink-api -sudo systemctl start interlink-virtual-kubelet - -# Remote services (on the plugin server) -sudo systemctl daemon-reload -sudo systemctl enable interlink-remote-plugin -sudo systemctl start interlink-remote-plugin - -# Check service status -sudo systemctl status interlink-tunnel -sudo systemctl status interlink-api -sudo systemctl status interlink-virtual-kubelet -``` - -### Service Operations - -Common systemd operations for managing tunneled interLink services: - -```bash -# View service logs -sudo journalctl -u interlink-tunnel -f -sudo journalctl -u interlink-api -f -sudo journalctl -u interlink-virtual-kubelet -f - -# Restart tunnel (will cascade to dependent services) -sudo systemctl restart interlink-tunnel - -# Stop all local interLink services -sudo systemctl stop interlink-virtual-kubelet interlink-api interlink-tunnel - -# Start all local interLink services -sudo systemctl start interlink-tunnel interlink-api interlink-virtual-kubelet - -# Check service dependencies -sudo systemctl list-dependencies interlink-virtual-kubelet -``` - -### Monitoring and Health Checks - -Create a comprehensive health check script for tunneled deployment: - -```bash title="/opt/interlink/bin/tunneled-health-check.sh" -#!/bin/bash - -# Health check script for tunneled interLink deployment -LOG_FILE="/opt/interlink/logs/health-check.log" -SOCKET_PATH="/tmp/interlink.sock" -REMOTE_HOST="remote-server" -REMOTE_PORT="4000" - -echo "$(date): Starting tunneled deployment health check" >> "$LOG_FILE" - -# Check SSH tunnel connectivity -if ! pgrep -f "ssh-tunnel" > /dev/null; then - echo "$(date): ERROR - SSH tunnel process not running" >> "$LOG_FILE" - exit 1 -fi - -# Check if Unix socket exists and is responding -if [ -S "$SOCKET_PATH" ]; then - response=$(curl -s --unix-socket "$SOCKET_PATH" http://unix/pinglink 2>/dev/null) - if [ $? -eq 0 ]; then - echo "$(date): Local API health check passed - $response" >> "$LOG_FILE" - else - echo "$(date): ERROR - Local API not responding via socket" >> "$LOG_FILE" - exit 1 - fi -else - echo "$(date): ERROR - Unix socket not found at $SOCKET_PATH" >> "$LOG_FILE" - exit 1 -fi - -# Check Virtual Kubelet node status -if kubectl get node interlink-node --no-headers 2>/dev/null | grep -q Ready; then - echo "$(date): Virtual Kubelet node is Ready" >> "$LOG_FILE" -else - echo "$(date): WARNING - Virtual Kubelet node not Ready" >> "$LOG_FILE" -fi - -# Test remote connectivity through tunnel -if nc -z -w5 127.0.0.1 4000 2>/dev/null; then - echo "$(date): Remote plugin connectivity through tunnel - OK" >> "$LOG_FILE" -else - echo "$(date): WARNING - Cannot reach remote plugin through tunnel" >> "$LOG_FILE" -fi - -echo "$(date): Tunneled deployment health check completed" >> "$LOG_FILE" -exit 0 -``` - -```bash -# Make executable -sudo chmod +x /opt/interlink/bin/tunneled-health-check.sh -sudo chown interlink:interlink /opt/interlink/bin/tunneled-health-check.sh -``` - -Create systemd timer for health checks: - -```ini title="/etc/systemd/system/interlink-tunneled-health-check.service" -[Unit] -Description=interLink Tunneled Health Check -After=interlink-virtual-kubelet.service -Requires=interlink-virtual-kubelet.service - -[Service] -Type=oneshot -User=interlink -Group=interlink -ExecStart=/opt/interlink/bin/tunneled-health-check.sh -``` - -```ini title="/etc/systemd/system/interlink-tunneled-health-check.timer" -[Unit] -Description=Run interLink Tunneled Health Check every 5 minutes -Requires=interlink-tunneled-health-check.service - -[Timer] -OnCalendar=*:0/5 -Persistent=true - -[Install] -WantedBy=timers.target -``` - -Enable the health check timer: - -```bash -sudo systemctl daemon-reload -sudo systemctl enable interlink-tunneled-health-check.timer -sudo systemctl start interlink-tunneled-health-check.timer -``` - -### Troubleshooting Tunneled Deployment - -#### SSH Tunnel Issues - -```bash -# Check SSH tunnel process -ps aux | grep ssh-tunnel - -# Test SSH connection manually -sudo -u interlink ssh -i /opt/interlink/.ssh/id_rsa interlink@remote-server - -# Check SSH tunnel logs -sudo journalctl -u interlink-tunnel --since "1 hour ago" - -# Test local socket -echo "test" | nc -U /tmp/interlink.sock -``` - -#### Virtual Kubelet Issues - -```bash -# Check Virtual Kubelet logs -sudo journalctl -u interlink-virtual-kubelet -f - -# Verify kubeconfig access -sudo -u interlink kubectl get nodes - -# Check node status -kubectl describe node interlink-node -``` - -#### Remote Plugin Issues - -```bash -# On remote server, check plugin status -sudo systemctl status interlink-remote-plugin - -# Check if plugin port is listening -netstat -tlnp | grep :4000 - -# Test plugin connectivity from remote server -curl -X GET http://localhost:4000/status -``` - -### Security Considerations for Tunneled Deployment - -#### SSH Security - -1. **Dedicated SSH keys**: Use separate keys for interLink tunneling -2. **Key restrictions**: Add restrictions in `authorized_keys`: - -```bash -# On remote server in ~/.ssh/authorized_keys -command="/usr/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding ssh-rsa AAAAB3... interlink-tunnel-key -``` - -3. **SSH configuration**: Secure SSH server configuration: - -```bash title="/etc/ssh/sshd_config.d/interlink.conf" -# Dedicated configuration for interLink tunnel user -Match User interlink - AllowTcpForwarding yes - AllowStreamLocalForwarding yes - PermitTunnel no - X11Forwarding no - AllowAgentForwarding no - PermitTTY no - ForceCommand /bin/false -``` - -#### Network Security - -```bash -# Firewall rules for local server -sudo ufw allow in on lo -sudo ufw allow out 22/tcp comment "SSH for tunnel" - -# Firewall rules for remote server -sudo ufw allow from to any port 22 comment "SSH tunnel" -sudo ufw allow 4000/tcp comment "Plugin API" -``` - -#### File Permissions - -```bash -# Secure SSH directory -sudo chmod 700 /opt/interlink/.ssh -sudo chmod 600 /opt/interlink/.ssh/id_rsa -sudo chmod 644 /opt/interlink/.ssh/id_rsa.pub /opt/interlink/.ssh/host_key - -# Secure configuration files -sudo chmod 640 /opt/interlink/config/* -sudo chown root:interlink /opt/interlink/config/* -``` - -This comprehensive tunneled deployment setup provides a robust, secure, and manageable solution for connecting Kubernetes clusters to remote compute resources through SSH tunneling. - -:::note - -For additional case studies and advanced configurations, reach out to the interLink community through the Slack channel. - -::: diff --git a/docs/docs/guides/08-systemd-deployment.mdx b/docs/docs/guides/08-systemd-deployment.mdx index 0c82eda66..eca2a9eee 100644 --- a/docs/docs/guides/08-systemd-deployment.mdx +++ b/docs/docs/guides/08-systemd-deployment.mdx @@ -91,6 +91,69 @@ PrivateTmp=true WantedBy=multi-user.target ``` +### SSH Tunnel Service + +Only needed for the [in-cluster deployment with a remote plugin](../cookbook/2-incluster.mdx#plugin-on-the-remote-system), +where the API server reaches its plugin through an SSH tunnel terminating on a +local Unix socket. Skip this if the plugin runs locally. + +```ini title="/etc/systemd/system/interlink-tunnel.service" +[Unit] +Description=interLink SSH Tunnel +After=network.target +Wants=network.target + +[Service] +Type=simple +User=interlink +Group=interlink +WorkingDirectory=/opt/interlink +ExecStart=/opt/interlink/bin/ssh-tunnel \ + -addr "remote-server:22" \ + -user "interlink" \ + -keyfile "/opt/interlink/.ssh/id_rsa" \ + -lsock "/tmp/interlink.sock" \ + -rport "4000" \ + -hostkeyfile "/opt/interlink/.ssh/host_key" + +Restart=always +RestartSec=10 +StandardOutput=append:/opt/interlink/logs/ssh-tunnel.log +StandardError=append:/opt/interlink/logs/ssh-tunnel.log + +# Security settings +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +ReadWritePaths=/opt/interlink/logs /tmp +PrivateTmp=true + +[Install] +WantedBy=multi-user.target +``` + +The SSH key material lives under `/opt/interlink/.ssh` and must be readable only +by the service user: + +```bash +sudo mkdir -p /opt/interlink/.ssh +sudo cp ~/.ssh/interlink_rsa /opt/interlink/.ssh/id_rsa +sudo cp ~/.ssh/interlink_host_key /opt/interlink/.ssh/host_key +sudo chown -R interlink:interlink /opt/interlink/.ssh +sudo chmod 700 /opt/interlink/.ssh +sudo chmod 600 /opt/interlink/.ssh/id_rsa +sudo chmod 644 /opt/interlink/.ssh/host_key +``` + +When the tunnel is in use, make the API server depend on it so systemd starts +them in the right order and restarts the API server if the tunnel is bounced: + +```ini title="/etc/systemd/system/interlink-api.service.d/tunnel.conf" +[Unit] +After=interlink-tunnel.service +Requires=interlink-tunnel.service +``` + ### InterLink API Server Service Create the InterLink API server systemd service: diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx new file mode 100644 index 000000000..2b1b6f53c --- /dev/null +++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx @@ -0,0 +1,436 @@ +--- +title: "SSH Tunnel Configuration" +description: "Reaching services inside offloaded pods on air-gapped HPC sites over an SSH login node" +sidebar_position: 14 +--- + +# SSH Tunnel Configuration + +Many HPC sites give compute nodes no outbound internet access, and many Kubernetes +clusters have no publicly reachable ingress. The [wstunnel](./10-wstunnel-configuration.mdx) +shadow needs both: the workload dials out to an ingress and runs a wstunnel client +there. On an air-gapped site it cannot. + +The SSH shadow inverts the direction. Instead of the compute node dialing out, the +cluster dials **in** to the site's SSH login node and forwards each exposed port +from the compute node the job landed on: + +``` +browser → Ingress / Service → shadow pod (ssh -L) → HPC login node → compute node (Jupyter) +``` + +The only requirement is outbound SSH from the cluster to the login node. The +offloaded pod runs nothing on its side: no wstunnel client, no WireGuard +configuration, no pre-exec injection. + +:::info +The SSH shadow replaces **wstunnel**, not [full mesh](./13-mesh-network-configuration.mdx). +It exposes the offloaded pod's ports to the cluster; it does not give the pod access +back into the cluster, so an offloaded workload still cannot reach in-cluster object +storage or message buses. Combining the two is rejected at startup and tracked in +[#548](https://github.com/interlink-hq/interLink/issues/548). +::: + +--- + +## Before you start: what the login node has to allow + +The shadow is an ordinary SSH client, so everything it needs is decided by the +login node's `sshd_config`. + +1. **`AllowTcpForwarding yes`** for the account the shadow logs in as. This is the + one that catches people out: plenty of HPC sites set `AllowTcpForwarding no` + globally and re-enable it only for a subset of users, often only those who + authenticate with MFA. Check it before anything else: + + ```bash + # from your own machine, with the same key/principal the shadow will use + ssh -N -L 19999::22 @ & + nc -z 127.0.0.1 19999 && head -c 40 < /dev/tcp/127.0.0.1/19999 + ``` + + If the site refuses, the shadow still starts and still reports Ready — the + failure only shows up when traffic arrives, as a connection reset at the client + and this line in `kubectl logs -c ssh-forward`: + + ``` + channel 1: open failed: administratively prohibited: open failed + ``` + + If the site will not allow it, switch to + [`ForwardMode: exec`](#forward-modes), which relays through a command on the + login node instead and needs no forwarding privilege. + +2. **A route from the login node to the compute nodes**, on the ports the pod + exposes. `ssh -L` resolves and connects to the compute node *from the login + node*, using whatever name the plugin reported. Verify with the name the plugin + actually reports (`hostname -f` on the compute node for the Slurm plugin), not + the short name: + + ```bash + ssh @ "curl -sv http://:/" + ``` + +3. **Outbound SSH from the cluster** to the login node's port, from the namespaces + shadows are created in. Network policies that restrict pod egress have to allow + it. + +`ssh -L` carries TCP only. UDP ports on the offloaded pod are skipped, with a +warning in the virtual kubelet log. + +--- + +## Which shadow does what + +| | Direction | Trigger | Workload must run | +| --- | --- | --- | --- | +| `wstunnel` (default) | cluster → pod | `EnableTunnel` + exposed ports | wstunnel client, dials out | +| `ssh` | cluster → pod | `EnableTunnel` + exposed ports | nothing | +| full mesh | bidirectional | `FullMesh: true`, every pod | `mesh.sh` (slirp4netns + WireGuard) | + +--- + +## Configuration + +### Virtual Kubelet + +```yaml +# VirtualKubeletConfig.yaml +Network: + EnableTunnel: true + ShadowMode: ssh + SSH: + LoginHost: login.hpc.example.org + User: alice + KeySecret: hpc-ssh-key +``` + +`EnableTunnel` turns shadow pods on for offloaded pods with exposed ports; +`ShadowMode` picks which shadow is rendered. Both are needed. + +### Helm + +```yaml +virtualNode: + network: + enableTunnel: true + shadowMode: ssh + ssh: + loginHost: login.hpc.example.org + user: alice + keySecret: hpc-ssh-key +``` + +See [`examples/ssh_tunnel.yaml`](https://github.com/interlink-hq/interlink-helm-chart/blob/main/interlink/examples/ssh_tunnel.yaml) +in the chart repository for a complete deployment. + +### Options + +| Option | Default | Description | +| --- | --- | --- | +| `LoginHost` | — | SSH login node to forward through (**required**) | +| `User` | — | Login name on that node (**required**) | +| `Port` | `22` | Login node's SSH port | +| `Image` | `ghcr.io/interlink-hq/interlink/ssh-tunnel:` | Image the shadow runs. Needs an ssh client, plus `kinit` for Kerberos | +| `Auth` | `publickey` | `publickey` or `kerberos` | +| `ForwardMode` | `portforward` | `portforward` (`ssh -L`) or `exec` (relay through a command on the login node) | +| `ExecConnectCommand` | `nc` | Command run on the login node in `exec` mode, invoked as ` ` | +| `KeySecret` | — | Secret holding the private key (`publickey`) | +| `KeySecretKey` | `id_ed25519` | Key inside `KeySecret` | +| `KeytabSecret` | — | Secret holding the keytab (`kerberos`) | +| `KeytabSecretKey` | `user.keytab` | Key inside `KeytabSecret` | +| `Principal` | — | Kerberos principal (`kerberos`) | +| `Krb5ConfigMap` | — | ConfigMap with a `krb5.conf`, mounted at `/etc/krb5.conf` | +| `KnownHostsConfigMap` | — | ConfigMap with a `known_hosts` file | +| `ReplicateCredentials` | `true` | Copy the credential into each shadow's namespace | +| `NodeWaitTimeout` | `2h` | How long the shadow waits for the compute node | +| `ExtraOptions` | `[]` | Extra ssh options, each passed as `-o