runlore runs without a CiliumNetworkPolicy (zero-trust gap)

[Smana]

What

The runlore workload has no CiliumNetworkPolicy at all:

$ kubectl get cnp -n runlore
No resources found in runlore namespace.

Every other pod-running workload on the platform ships one. The constitution mandates default-deny + explicit allow for anything that runs a pod, so this is a straight compliance gap: runlore's egress is currently unrestricted.

How it surfaced

While chasing a suspected AWS authorization problem with runlore — there wasn't one. The EPI is Ready, the PodIdentityAssociation is Ready, the IAM policy is attached and correctly scoped read-only (ec2:Describe*, eks:Describe*, autoscaling:Describe*, cloudtrail:LookupEvents), and there are no AccessDenied errors in its logs.

The irony is worth writing down: runlore's AWS access works because nothing is restricting its egress. The moment a policy is added it will break unless the rules below are right — so this needs writing carefully, not copy-pasting.

What the policy has to allow

Egress, at minimum:

• kube-dns — and the L7 DNS rule (toPorts.rules.dns.matchPattern: "*") is mandatory. Without it Cilium proxies the query but never sees the response IPs, so any toFQDNs allowlist has no IPs to match: DNS keeps working while every follow-up TCP connection is silently dropped (trap refacto: split network and eks tofu states #1 in .claude/rules/cilium-network-policies.md).
• EKS Pod Identity agent at 169.254.170.23:80 — the one that will bite. The agent runs on the node's host network, so Cilium classifies the destination as the host entity: toCIDR: 169.254.170.23/32 alone silently fails, and toEntities: world does not cover link-local. It needs toEntities: ["host"] scoped to TCP 80 (trap chore(terraform): use the subnet router module v1.0.3 #3). Symptom if wrong: Connect timeout on endpoint URL: 'http://169.254.170.23/v1/credentials' — i.e. precisely the "runlore isn't authorized in AWS" failure this issue was mistaken for.
• Hubble Relay (kube-system) — runlore dials it for network flows (chore(deps): update helm release prometheus-operator-crds to v13.0.1 #321 pinned its TLS floor to 1.2).
• Kubernetes API — its read-only cluster tooling.
• world:443 — LLM provider, Slack, GitHub (the KB repo). toFQDNs is a maintenance chase here because matchPattern: "*" does not span dots (trap feat(tofu): use the tailscale provider #2), so scope this deliberately rather than by reflex.

Ingress: the Cilium Gateway data-plane pods terminating the Tailscale-fronted HTTPRoute are the ingress entity, not a pod selector.

Acceptance

• CiliumNetworkPolicy in observability/base/runlore/, default-deny + explicit allow
• After it applies, runlore still reaches AWS (pod identity), Hubble, the API server, the LLM and Slack — verified with hubble observe --pod runlore/runlore-0 --verdict DROPPED, not by "the Kustomization reconciled"
• An investigation runs end to end with the policy in place

Found during a platform validation session on feat/flux-schema-validation.

[Smana]

Correcting the premise: runlore is not unprotected. The gap is real but much narrower than this issue describes.

Verified on the live cluster (2026-08-02).

What the issue got wrong

The finding rests on:

$ kubectl get cnp -n runlore
No resources found in runlore namespace.

That command only lists CiliumNetworkPolicies. runlore's policy is a standard networking.k8s.io/v1 NetworkPolicy, rendered by the chart — which Cilium enforces just the same:

$ kubectl get netpol -n runlore
NAME      POD-SELECTOR                                                        AGE
runlore   app.kubernetes.io/instance=runlore,app.kubernetes.io/name=runlore   3h10m

$ kubectl get ciliumnetworkpolicies.cilium.io -n runlore
NAME                       AGE    VALID
runlore-aws-pod-identity   162m   True

(The second one is missing from kubectl get cnp output only because the short name cnp resolves to a different resource here — note the Warning: short name "cnp" could also match lower priority resource ciliumnetworkpolicies.cilium.io. Worth using the fully-qualified name in future checks.)

So "runlore's egress is currently unrestricted" is not accurate, and the corollary — "the moment a policy is added it will break" — does not apply: the policy is already on and runlore works.

Both are already configured in observability/base/runlore/helmrelease.yaml under values.networkPolicy (enabled: true, awsPodIdentity: true, plus an extraEgress for the VictoriaMetrics/VictoriaLogs ports).

The actual posture today

policyTypes: [Ingress, Egress], so everything not listed is denied:

Direction	Allowed	Verdict
Egress 53	kube-dns pods in kube-system only	✅
Egress 8080	runlore peer pods (leader forwarding)	✅
Egress 8428 / 9428	observability namespace	✅
Egress 80	toEntities: [host] — the EKS Pod Identity agent, via the Cilium policy	✅ correct, and exactly the trap #3 the issue warned about
Egress 443 + 6443	any destination	⚠️ the real gap
Ingress 8080	any source (no from:)	⚠️ second gap, unmentioned by the issue

Two of the issue's stated requirements turn out to be moot:

• Hubble Relay — not needed. The network source is deferred in config (# network: deferred (needs a Hubble/flow-log provider choice)), so nothing dials Hubble and 4245 is correctly absent.
• L7 DNS matchPattern: "*" — only required to make toFQDNs work. There is no toFQDNs rule today, so nothing is silently dropping.

What is genuinely worth fixing

1. Ingress is unscoped (cheap, no design question). Anything in the cluster can reach :8080. The chart already supports networkPolicy.ingressFrom, and it deliberately keeps peer-to-peer reachable when you narrow it. Scoping to the Alertmanager/VMAlert namespace plus the Tailscale gateway data plane is a values-only change.

2. Egress 443 to the world (the judgment call). This is what .claude/rules/cilium-network-policies.md trap #4 says to avoid on a long-lived serving pod. The destination set is bounded and knowable:

• api.z.ai — the LLM (GLM via the OpenAI-compatible wire protocol)
• github.com, api.github.com — the KB repo and the GitHub App
• Slack — outbound API / webhook
• AWS regional endpoints — ec2, eks, autoscaling, cloudtrail, sts in eu-west-3
• the Kubernetes API on 6443

The honest counter-argument, and why this needs a decision rather than a reflex: git clone against GitHub also reaches codeload.github.com and object hosts, and AWS SDK endpoint sets drift — and per trap #2, matchPattern: "*" does not span dots, so each subdomain level needs its own pattern. The chart's strict: true mode allowlists by CIDR, which is worse than FQDNs for SaaS.

So the choice is: a Cilium toFQDNs policy (requires adding the mandatory L7 DNS rule — trap #1 — and accepting some ongoing maintenance), or keeping 443-to-world with the rationale written down.

Suggested retitle

runlore: scope NetworkPolicy ingress, and decide on tightening egress 443 — the current title asserts a zero-trust gap that does not exist as stated.

[Smana]

Ingress scoping shipped and verified live — 809536a7 on chore/runlore-v0.12.0. Egress 443 left as-is per the decision above; that half of the issue stays open.

What changed

Two policies, because one alone could not do it:

• values.networkPolicy.ingressFrom → observability namespace. The chart always prepends a peer podSelector, so leader forwarding (runlore#264) survives.
• New observability/base/runlore/ciliumnetworkpolicy-ingress.yaml → fromEntities: [ingress] on 8080.

The second one is the part that would have bitten. Setting ingressFrom alone black-holes every externally routed request: runlore's HTTPRoute parents onto platform-public (GatewayClass cilium), and this cluster runs the shared cilium-envoy DaemonSet with hostNetwork=true, so that traffic arrives as Cilium's reserved ingress entity — no namespaceSelector can ever match it. This is the ingress-side counterpart of trap #3, and it is exactly what the issue's own closing line predicted. Cilium unions the two policies, so the effective allow-list is observability + peers + ingress, and nothing else.

Verification

Not "it reconciled" — all three ingress paths exercised after the policy was live:

Path	Evidence
Gateway → pod (Slack)	POST https://runlore.cloud.ogenki.io/slack/interactions → 401 unauthorized in 50ms. That is runlore's own signature rejection, so the request traversed ELB → cilium-envoy → pod. (A GET /healthz gives 404 from Envoy and proves nothing — only POST /slack/interactions is routed.)
observability → pod (scrape)	up{namespace="runlore"} = 1 for both pods, fresh timestamp
peer → peer	both pods 1/1 Running, 0 restarts, leader election intact
Drops	hubble observe --pod runlore/runlore-1 --verdict DROPPED --last 30 → no rows

./scripts/validate-manifests.sh → Valid: 1191, Invalid: 0, Skipped: 0, all gates passed, exit 0.

Remaining scope

Egress 443/6443 to any destination. Deliberately unchanged — see the trade-off in the previous comment. Retitling to something like runlore: tighten egress 443 to toFQDNs would now match what is actually left.