kaput-not

A CNI-agnostic Kubernetes controller for managing Netmaker egress rules

Note: "kaput-not" is a wordplay on the German word "kaputt" (broken), suggesting this controller keeps the overlay network from breaking.

The Problem

When running Kubernetes on WireGuard-based mesh networks like Netmaker, your network will eventually break without special preparation. The issue: private IPv4 addresses from pod CIDR ranges leak into WireGuard tunnels as endpoint IP addresses, but WireGuard endpoints must be publicly routable IPs.

Symptom: When the network breaks, running sudo wg show all endpoints on each node will show private IP addresses from peer node's pod CIDR ranges instead of their public IPv4 addresses.

Triggers: Common operations can break the network unexpectedly:

Running netclient pull in a terminal on a node
Clicking "Sync Device" or "Sync All" in the Netmaker dashboard
Interface up/down events (network hiccups, node reboots, etc.)

The Solution: kaput-not automatically manages Netmaker egress rules to prevent this leakage by adding pod CIDR ranges to the WireGuard allowed-ips for each peer. This ensures pod traffic stays within the mesh and never gets mistaken for tunnel endpoints.

Overview

kaput-not is a Kubernetes controller that automatically synchronizes pod CIDR allocations from Kubernetes nodes to Netmaker Egress gateway rules. It works with any CNI that populates the standard spec.podCIDRs field on Node resources (Cilium, Calico, Flannel, etc.).

Key Features

✅ CNI-Agnostic: Works with any CNI using standard Kubernetes API
✅ Multi-Cluster Safe: Optional cluster name scoping prevents cross-cluster egress rule conflicts
✅ Leader Election: High availability with multiple replicas
✅ Rebuild Resilient: Uses index-based lookup for stable egress rule management
✅ Multi-CIDR Support: Handles multiple pod CIDRs per node (IPv4/IPv6 dual-stack)
✅ Minimal Permissions: Read-only access to nodes, lease management only
✅ Twelve-Factor App: Configuration via environment variables
✅ Small Footprint: <20MB Docker image using distroless

How It Works

Watches Kubernetes Node resources for pod CIDR changes
Maps Kubernetes node names to Netmaker node IDs
Syncs pod CIDRs to Netmaker Egress rules
Maintains stable egress rules using index-based lookup

Egress Rule Format

Each pod CIDR gets its own egress rule with:

Description: Managed by kaput-not (DO NOT EDIT): index=0 (stable identifier, or cluster=us-east index=0 for multi-cluster)
Name: node-name pods (1/2) (human-friendly)
Range: Pod CIDR value (e.g., 10.160.0.0/24)
NAT: false (no source NAT for pod CIDRs)
Nodes: Map containing the Netmaker node UUID (e.g., {"uuid": 500})

The index-based description combined with the node ID in the nodes map ensures that egress rules survive pod CIDR changes while preventing orphaned rules.

Multi-Cluster Support

When multiple Kubernetes clusters share a single Netmaker network, use cluster name scoping to prevent conflicts:

Single-cluster mode (default): Leave clusterName empty - manages all kaput-not egress rules
Multi-cluster mode: Set clusterName to a unique identifier (e.g., us-east) - only manages egress rules with that cluster name

Egress rules include the cluster name in the description: Managed by kaput-not (DO NOT EDIT): cluster=us-east index=0

Migration safety: When transitioning from single-cluster to multi-cluster mode, existing egress rules without cluster names are left untouched and new egress rules with cluster names are created.

Installation

Prerequisites

Kubernetes cluster (1.19+)
Helm 3.0+
Netmaker instance with API access
Netmaker service account (see setup below)

Setting Up Netmaker Authentication

Important: kaput-not requires Netmaker user credentials (username/password), not enrollment keys.

Creating a Service Account

Create a dedicated Netmaker user for kaput-not:

Option 1: Via Netmaker Dashboard (Recommended)

Log into Netmaker Dashboard at https://dashboard.netmaker.example.com
Go to Users → Add User
Create user:
- Username: kaput-not
- Password: Generate strong password (e.g., openssl rand -base64 32)
- Role: Grant node management and egress operations permissions
Save credentials securely

Option 2: Via Netmaker API

# Authenticate as admin
ADMIN_TOKEN=$(curl -s -X POST https://api.netmaker.example.com/api/users/adm/authenticate \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "admin-password"}' \
  | jq -r '.Response.AuthToken')

# Create service account
curl -X POST https://api.netmaker.example.com/api/users \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "kaput-not",
    "password": "strong-random-password"
  }'

Required Permissions

The service account must have permissions to:

Read node information
List, create, update, and delete egress gateways for the network

Security Best Practices

✅ Use dedicated service account (don't reuse admin credentials)
✅ Generate strong random passwords (32+ characters)
✅ Limit permissions to only node and egress management
✅ Store credentials in Kubernetes Secrets (never commit to git)
✅ Rotate credentials periodically

Quick Start

Option 1: Install from GHCR (Recommended)

helm install kaput-not oci://ghcr.io/bsure-analytics/charts/kaput-not \
  --namespace kube-system \
  --create-namespace \
  --set netmaker.apiUrl="https://api.netmaker.example.com" \
  --set netmaker.username="kaput-not" \
  --set netmaker.password="your-password-here"

Option 2: Install from Local Chart

helm install kaput-not ./charts/kaput-not \
  --namespace kube-system \
  --create-namespace \
  --set netmaker.apiUrl="https://api.netmaker.example.com" \
  --set netmaker.username="kaput-not" \
  --set netmaker.password="your-password-here"

Multi-Cluster Deployment

If multiple Kubernetes clusters share a Netmaker network, set a unique cluster name:

helm install kaput-not oci://ghcr.io/bsure-analytics/charts/kaput-not \
  --namespace kube-system \
  --create-namespace \
  --set netmaker.apiUrl="https://api.netmaker.example.com" \
  --set netmaker.username="kaput-not" \
  --set netmaker.password="your-password-here" \
  --set clusterName="us-east"  # Unique identifier for this cluster

Recommended: Use a separate values file for credentials (never commit to git):

# Create my-values.yaml with your configuration
cat > my-values.yaml <<EOF
netmaker:
  apiUrl: "https://api.netmaker.example.com"
  username: "kaput-not"
  password: "your-secure-password"
EOF

# Install from GHCR using the values file
helm install kaput-not oci://ghcr.io/bsure-analytics/charts/kaput-not \
  --namespace kube-system \
  --create-namespace \
  -f my-values.yaml

# Or install from local chart
helm install kaput-not ./charts/kaput-not \
  --namespace kube-system \
  --create-namespace \
  -f my-values.yaml

Verify deployment:

kubectl get pods -n kube-system -l app.kubernetes.io/name=kaput-not
kubectl logs -n kube-system -l app.kubernetes.io/name=kaput-not --tail=50

Check which pod is the leader:

kubectl get lease -n kube-system kaput-not -o yaml

Configuration

Configuration is managed via Helm values. See charts/kaput-not/values.yaml for all available options.

Required Values

netmaker.apiUrl: Netmaker API endpoint (e.g., https://api.netmaker.example.com)
netmaker.username: Netmaker username for authentication
netmaker.password: Netmaker password for authentication

Networks are auto-discovered from the Netmaker API based on which networks each Kubernetes host participates in.

Common Optional Values

clusterName: Unique cluster identifier for multi-cluster deployments (default: "" for single-cluster mode)
replicaCount: Number of controller replicas (default: 2)
image.repository: Docker image repository (default: ghcr.io/bsure-analytics/kaput-not)
image.tag: Docker image tag (default: chart appVersion)
leaderElection.enabled: Enable leader election (default: true)
priorityClassName: Priority class for pod scheduling (default: system-cluster-critical)
tolerations: Pod tolerations (default: tolerates control-plane nodes)
resources.requests/limits: CPU and memory resources

Example: Custom Configuration

# Using GHCR
helm upgrade kaput-not oci://ghcr.io/bsure-analytics/charts/kaput-not \
  --namespace kube-system \
  --set replicaCount=3 \
  --set resources.requests.memory="128Mi" \
  --set image.tag="v1.0.0"

# Or using local chart
helm upgrade kaput-not ./charts/kaput-not \
  --namespace kube-system \
  --set replicaCount=3 \
  --set resources.requests.memory="128Mi" \
  --set image.tag="v1.0.0"

Environment Variables (Local Development)

For local development without Helm:

Required:

KUBECONFIG: Path to kubeconfig (for local development)
NETMAKER_API_URL: Netmaker API endpoint
NETMAKER_USERNAME: Netmaker username
NETMAKER_PASSWORD: Netmaker password

Networks are auto-discovered from the Netmaker API based on which networks each Kubernetes host participates in.

Optional:

K8S_CLUSTER_NAME: Cluster identifier for multi-cluster deployments (empty = single-cluster mode)
LEADER_ELECTION_ENABLED: Enable leader election (auto-detected: disabled for local dev, enabled in-cluster)
LEADER_ELECTION_NAMESPACE: Namespace for lease resource (auto-detected: pod's namespace in-cluster, kube-system for local dev)
LEADER_ELECTION_ID: Lease resource name (default: kaput-not)

Architecture

kaput-not follows Hexagonal Architecture (Ports & Adapters):

cmd/kaput-not/          # CLI adapter ("let it crash")
  ├── main.go           # Entry point, panics on errors
  └── config.go         # Environment variable loading

pkg/                    # Library (pure business logic)
  ├── netmaker/         # Netmaker API client with TTL-based caching
  ├── reconciler/       # Reconciliation logic
  ├── controller/       # Kubernetes controller (informer)
  └── leaderelection/   # Leader election logic

charts/kaput-not/       # Helm chart
  ├── Chart.yaml        # Chart metadata
  ├── values.yaml       # Default configuration values
  └── templates/        # Kubernetes resource templates

Design Principles

KISS: No retry logic, no state persistence, let it crash and restart
DRY: Shared reconciliation function for CREATE/UPDATE
Twelve-Factor: Configuration via environment only
Library + Adapter: Business logic returns errors, CLI adapter panics
Intelligent Caching: TTL-based caching (30 second default) reduces API calls to Netmaker while maintaining consistency

Caching Layer

kaput-not includes a TTL-based caching layer that significantly reduces API calls to Netmaker:

Default TTL: 30 seconds for all cached responses
What's cached: Authentication tokens, host lookups, node lookups, and egress gateway lists
Network-aware: Separate cache entries per Netmaker network
Thread-safe: Uses mutex locks for concurrent access
Auto-invalidation: Expires on TTL timeout and authentication failures
Transparent: No code changes needed - caching happens automatically in the HTTP client

This reduces load on the Netmaker API while maintaining near real-time consistency, especially important during the periodic 10-minute resync cycles.

Multi-Network Support

kaput-not automatically discovers and manages Netmaker networks for each Kubernetes node:

Auto-discovery: Networks are automatically detected from the Netmaker API based on which networks each host participates in
Multi-network capability: A single Kubernetes node can participate in multiple Netmaker networks simultaneously
Independent egress rules: Each network gets separate egress rules with index-based management
Network-aware caching: Cache is network-scoped to prevent cross-network data leakage
No manual configuration: No need to specify network names - everything is discovered automatically

High Availability

kaput-not supports high availability through Kubernetes lease-based leader election:

2 replicas (configurable via deployment)
Only one active controller at a time
Automatic failover if leader fails
No split-brain due to lease-based locking
Automatic rolling updates on configuration changes via ConfigMap/Secret checksums

Configuration Updates

The Helm chart automatically triggers rolling updates when configuration changes:

ConfigMap and Secret checksums are included in pod template annotations
Changing values like clusterName, netmaker.apiUrl, or netmaker.password triggers zero-downtime rolling updates
No need to manually restart pods after configuration changes

Check which replica is the leader:

kubectl get lease -n kube-system kaput-not -o yaml

Resource Requirements and Scaling

kaput-not has O(n) memory complexity where n is the number of Kubernetes nodes.

Memory Usage by Cluster Size

Cluster Size	Memory Request	Memory Limit	Notes
< 500 nodes	64 Mi	128 Mi	Default configuration
500-1,000 nodes	128 Mi	256 Mi	Recommended for medium clusters
1,000-5,000 nodes	256 Mi	512 Mi	Recommended for large clusters
5,000+ nodes	512 Mi	1 Gi	Recommended for very large clusters

What Drives Memory Usage

The primary memory consumer is the Kubernetes informer cache, which stores a complete copy of all Node objects:

Per-node overhead: ~25 KB (metadata + spec + status)
Workqueue: Negligible (transient, usually empty)
Application code: ~10-50 MB (Go runtime + libraries)

Example: A 100-node cluster uses approximately 50-100 MB total memory.

Adjusting Resource Limits

Update resource limits in your Helm values:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    cpu: "200m"

Event Processing

The controller provides both real-time and periodic reconciliation:

✅ Immediate response to Node events (add/update/delete via Kubernetes watch API)
✅ Full reconciliation on startup (syncs all existing nodes)
✅ Periodic resync every 10 minutes (drift correction, no action if pod CIDRs unchanged)

This ensures consistency after downtime and corrects any manual changes to Netmaker egress rules.

Local Development

Prerequisites

Go 1.25+
Docker (for building images)
kubectl and access to a Kubernetes cluster

Building

# Build binary
make build

# Run tests
make test

# Run linters
make lint

# Build Docker image
make docker-build

# Lint Helm chart
make helm-lint

# Package Helm chart
make helm-package

Running Locally

Set environment variables and run:

export NETMAKER_API_URL="https://api.netmaker.example.com"
export NETMAKER_USERNAME="kaput-not"
export NETMAKER_PASSWORD="your-password"
export KUBECONFIG="$HOME/.kube/config"

# Networks are auto-discovered from Netmaker API
./bin/kaput-not

Note: Leader election is automatically disabled when running locally (not in-cluster).

Troubleshooting

Authentication failures

Symptom: Logs show authentication failed with HTTP status 401

Solutions:

Verify credentials in Secret:

kubectl get secret kaput-not -n kube-system -o jsonpath='{.data.NETMAKER_USERNAME}' | base64 -d

Test credentials manually:

curl -X POST https://api.netmaker.example.com/api/users/adm/authenticate \
  -H "Content-Type: application/json" \
  -d '{
    "username": "kaput-not",
    "password": "your-password"
  }'
# Should return AuthToken if valid

Verify user exists in Netmaker Dashboard → Users

If password changed, upgrade Helm release with new password:

# Using GHCR
helm upgrade kaput-not oci://ghcr.io/bsure-analytics/charts/kaput-not \
  --namespace kube-system \
  --reuse-values \
  --set netmaker.password="<new-password>"

# Or using local chart
helm upgrade kaput-not ./charts/kaput-not \
  --namespace kube-system \
  --reuse-values \
  --set netmaker.password="<new-password>"

Controller not starting

# Check logs
kubectl logs -n kube-system -l app.kubernetes.io/name=kaput-not --tail=100

# Common issues:
# 1. Wrong configuration - verify Helm values
# 2. Wrong API URL - check netmaker.apiUrl value
# 3. Authentication failed - see "Authentication failures" above

Egress rules not created

# Check if node has pod CIDRs assigned
kubectl get node <node-name> -o jsonpath='{.spec.podCIDRs}'

# Check controller logs for reconciliation errors
kubectl logs -n kube-system -l app.kubernetes.io/name=kaput-not | grep "failed to reconcile"

# Verify Netmaker host exists with matching name
curl -H "Authorization: Bearer $TOKEN" \
  https://api.netmaker.example.com/api/hosts | jq '.[] | select(.name=="node-name")'

Multiple leaders / split-brain

# Check lease status
kubectl get lease -n kube-system kaput-not -o yaml

# Verify only one pod is active leader (check logs)
kubectl logs -n kube-system -l app.kubernetes.io/name=kaput-not | grep "Became leader"

Pod CIDR changed but egress not updated

The controller should detect this automatically. If not:

# Force reconciliation by restarting the controller
kubectl rollout restart deployment -n kube-system -l app.kubernetes.io/name=kaput-not

# Check if the egress rule exists with old CIDR
curl -H "Authorization: Bearer $TOKEN" \
  "https://api.netmaker.example.com/api/v1/egress?network=development" | \
  jq '.Response[] | select(.description | contains("Managed by kaput-not"))'

Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

Licensed under the Apache License, Version 2.0. See LICENSE file for details.

Published Artifacts

kaput-not releases are automatically published to GitHub Container Registry (GHCR):

Docker Images: ghcr.io/bsure-analytics/kaput-not
- View at: https://github.com/bsure-analytics/kaput-not/pkgs/container/kaput-not
- Tags: latest, version numbers (1.0.0, 1.0, 1), branch names, git SHA
Helm Charts: oci://ghcr.io/bsure-analytics/charts/kaput-not
- Published as OCI artifacts (Helm 3.8+ required)
- Version numbers match git tags (v1.0.0 → chart version 1.0.0)
- Install: helm install kaput-not oci://ghcr.io/bsure-analytics/charts/kaput-not

Related Projects

Netmaker - WireGuard mesh network platform

Name		Name	Last commit message	Last commit date
Latest commit History 22 Commits
.github/workflows		.github/workflows
charts/kaput-not		charts/kaput-not
cmd/kaput-not		cmd/kaput-not
pkg		pkg
.gitignore		.gitignore
.golangci.yaml		.golangci.yaml
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
renovate.json		renovate.json

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

kaput-not

The Problem

Overview

Key Features

How It Works

Egress Rule Format

Multi-Cluster Support

Installation

Prerequisites

Setting Up Netmaker Authentication

Creating a Service Account

Required Permissions

Security Best Practices

Quick Start

Option 1: Install from GHCR (Recommended)

Option 2: Install from Local Chart

Multi-Cluster Deployment

Configuration

Required Values

Common Optional Values

Example: Custom Configuration

Environment Variables (Local Development)

Architecture

Design Principles

Caching Layer

Multi-Network Support

High Availability

Configuration Updates

Resource Requirements and Scaling

Memory Usage by Cluster Size

What Drives Memory Usage

Adjusting Resource Limits

Event Processing

Local Development

Prerequisites

Building

Running Locally

Troubleshooting

Authentication failures

Controller not starting

Egress rules not created

Multiple leaders / split-brain

Pod CIDR changed but egress not updated

Contributing

License

Published Artifacts

Related Projects

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages