Support cross-namespace service dependencies for KubePlus Kind instances

[devdattakulkarni]

Summary

KubePlus Kinds are registered from arbitrary Helm charts, so KubePlus has no built-in
understanding of a chart's values.yaml schema. When an instantiated Kind needs to reach
a shared service running in a different namespace, KubePlus currently has no mechanism to
provision the networking (and RBAC, where relevant) required for that cross-namespace call —
this has to be wired by hand today. We provide kubectl plugins for this. But it will be better if this can be done as part of the Operator.

This issue proposes a KubePlus-owned annotation, kubeplus.io/cross-ns-deps, that a Kind
instance can carry to declare its cross-namespace dependencies. KubePlus reconciles the
annotation into the NetworkPolicy objects (and namespace labels) needed to allow the traffic,
independent of what the underlying chart actually does with those values.

Motivating example

• A MCPServer Kind is registered from a Helm chart and instantiated once, as
  shared-k8sgpt-mcp in namespace platform-mcp.
• An Agent Kind (wrapping a kagent Agent + ModelConfig) is registered from a second
  Helm chart, and instantiated twice, into two separate, isolated namespaces:
  • team-a-agent-instance → namespace team-a
  • team-b-agent-instance → namespace team-b
• Both team-a and team-b are otherwise network-isolated (default-deny), and neither
  should be able to reach the other. Both should be able to reach shared-k8sgpt-mcp in
  platform-mcp, and nothing else in platform-mcp beyond that one service.
• The Agent chart's values.yaml already has whatever fields it needs (e.g.
  mcpServer.name / mcpServer.namespace) to template the kagent Agent CR's
  namespace-qualified RemoteMCPServer reference. KubePlus is not involved in that part —
  it only needs to open the network path.

Annotation name and shape

Annotation key: kubeplus.io/cross-ns-deps

Placed on the KubePlus Kind instance CR (not on any resource rendered by the chart).
Value is a JSON array, allowing more than one dependency per instance from day one:

apiVersion: cloudark.io/v1
kind: Agent
metadata:
  name: team-a-agent-instance
  namespace: team-a
  annotations:
    kubeplus.io/cross-ns-deps: |
      [
        {
          "service": "shared-k8sgpt-mcp",
          "namespace": "platform-mcp",
          "port": 8089
        }
      ]
spec:
  mcpServer:
    name: shared-k8sgpt-mcp
    namespace: platform-mcp
  modelConfigRef: default-model-config
  systemMessage: "..."

Field definitions:

Field	Required	Meaning
service	yes	Name of the target Kubernetes Service (or Kind instance) being depended on
namespace	yes	Namespace the target lives in
port	no	If present, scope the NetworkPolicy egress/ingress rule to this port; if absent, allow all ports to the target namespace

Note: service/namespace here duplicate whatever the chart's own values.yaml uses
(e.g. spec.mcpServer.name/spec.mcpServer.namespace above) — this is intentional.
KubePlus's dependency wiring stays chart-agnostic; the chart author is responsible for
keeping the two in sync, since only they know how their chart's values.yaml maps to the
running Service.

How KubePlus should handle the annotation

On Kind instance create/update:

1. Watch for kubeplus.io/cross-ns-deps on any KubePlus Kind instance, across all
   registered Kinds (this is Kind-agnostic — it's not specific to the Agent or
   MCPServer Kind).
2. Parse the JSON array. For each entry:
   • Label the consumer namespace (the instance's own namespace) with
     kubeplus.io/consumes.<target-namespace>.<target-service>: "true"
     (or a hashed/short form of this if label value/key length limits are a concern —
     labels are capped at 63 chars).
   • Create or update a deterministically-named egress NetworkPolicy in the
     consumer's namespace: kubeplus-egress-<instance-name>-<target-service>.
     • podSelector: matches only the pods belonging to this instance (from the chart's
       standard instance/app labels), not the whole namespace — even though today
       each Agent instance has its own namespace, scoping to the instance's pods keeps
       the policy correct if a namespace ever hosts more than one instance later.
     • egress: allow to namespaceSelector matching kubernetes.io/metadata.name: <target-namespace>, restricted to port if given.
   • Ensure the target namespace carries a matching label so its own ingress policy
     can select on it, e.g. label the target namespace itself with
     kubeplus.io/shared-service: <target-service> (idempotent — set once, reused by
     every consumer).
   • Create or update one shared, deterministically-named ingress NetworkPolicy in
     the target namespace: kubeplus-ingress-<target-service>.
     • podSelector: matches the target service's pods.
     • ingress: allow from any namespace carrying the corresponding
       kubeplus.io/consumes.<target-namespace>.<target-service>: "true" label (via
       namespaceSelector), restricted to port if given.
     • This object is shared across all consumers of the same target — it is not
       rewritten per consumer; only the set of namespaces it selects grows as more
       consumer-namespace labels appear. This avoids read-modify-write races between
       concurrently reconciled Agent instances.
3. Requeue/reconcile on any change to the annotation value (new dependency added/removed).

On Kind instance delete:

1. Remove the consumer-namespace label(s)
   (kubeplus.io/consumes.<target-namespace>.<target-service>) that were set for this
   instance's dependencies. If other instances in the same namespace still declare the
   same dependency, leave the label in place (reference-count by checking for other
   instances with the same dependency before removing).
2. Delete the per-instance egress NetworkPolicy
   (kubeplus-egress-<instance-name>-<target-service>) — this one is always safe to
   delete unconditionally since it's scoped to the deleted instance's own pods.
3. Do not delete the shared ingress NetworkPolicy in the target namespace — it
   should persist as long as any consumer remains selected by it. If the label removal
   in step 1 empties the set of matching namespaces, the policy remains present but
   inert (matches nothing), which is fine; garbage-collecting the ingress policy object
   itself is optional cleanup, not correctness-critical.

RBAC note (separate from NetworkPolicy)

If the Kind's controller (or the underlying kagent controller) needs API-level read
access to the target object across namespaces (e.g. get/watch on the
RemoteMCPServer CR in platform-mcp), that's a second, smaller concern: a RoleBinding
in the target namespace granting the relevant ServiceAccount get/list/watch on that
specific object. Worth deciding whether this is in scope for kubeplus.io/cross-ns-deps
v1 or tracked as a follow-up — flagging it here so it isn't lost, but the NetworkPolicy
wiring above is the core ask.

Acceptance criteria

• kubeplus.io/cross-ns-deps annotation is parsed on any KubePlus Kind instance
  (Kind-agnostic).
• Creating an Agent instance with the annotation produces exactly one new egress
  NetworkPolicy in the agent's namespace, scoped to that instance's pods.
• Creating a second Agent instance (different namespace, same target dependency)
  results in the shared ingress NetworkPolicy in platform-mcp now selecting both
  consumer namespaces, without modifying/duplicating the egress policies of either
  agent.
• team-a and team-b cannot reach each other (neither declared a dependency on the
  other).
• Deleting one Agent instance removes only its own egress policy and its own
  namespace's consumption label; the other Agent instance's access to
  shared-k8sgpt-mcp is unaffected.
• Deleting the last consumer leaves the shared ingress policy present but matching no
  namespaces (no error state).

Demo steps

1. Set up cluster and register Kinds

   minikube start
   kubectl create namespace platform-mcp
   kubectl create namespace team-a
   kubectl create namespace team-b
   # Register the two Helm charts as KubePlus Kinds
   kubectl apply -f mcpserver-kind.yaml
   kubectl apply -f agent-kind.yaml

2. Instantiate the shared MCP server

   kubectl apply -f - <<EOF
   apiVersion: cloudark.io/v1
   kind: MCPServer
   metadata:
     name: shared-k8sgpt-mcp
     namespace: platform-mcp
   spec:
     image: k8sgpt-mcp:latest
     port: 8089
   EOF

   Verify the Service and pod come up in platform-mcp.

3. Instantiate the first Agent, in team-a, with the dependency annotation

   kubectl apply -f - <<EOF
   apiVersion: cloudark.io/v1
   kind: Agent
   metadata:
     name: team-a-agent-instance
     namespace: team-a
     annotations:
       kubeplus.io/cross-ns-deps: |
         [{"service":"shared-k8sgpt-mcp","namespace":"platform-mcp","port":8089}]
   spec:
     mcpServer:
       name: shared-k8sgpt-mcp
       namespace: platform-mcp
     modelConfigRef: default-model-config
   EOF

   Show:

   kubectl get networkpolicy -n team-a
   kubectl get networkpolicy -n platform-mcp
   kubectl get namespace team-a --show-labels
   kubectl get namespace platform-mcp --show-labels

   Point out the per-instance egress policy in team-a and the shared ingress policy in
   platform-mcp.

4. Instantiate the second Agent, in team-b, with the same dependency
   Repeat step 3 for team-b-agent-instance / namespace team-b. Re-check the ingress
   policy in platform-mcp — it should now select both team-a and team-b, without a
   second ingress policy object appearing.

5. Prove isolation

   # team-a can reach the shared MCP server
   kubectl exec -n team-a <agent-pod> -- curl -sS http://shared-k8sgpt-mcp.platform-mcp.svc.cluster.local:8089/mcp
   
   # team-a cannot reach team-b
   kubectl exec -n team-a <agent-pod> -- curl -m 3 http://<team-b-agent-svc>.team-b.svc.cluster.local  # should time out

6. Delete one Agent instance and show scoped cleanup

   kubectl delete agent team-a-agent-instance -n team-a
   kubectl get networkpolicy -n team-a          # egress policy gone
   kubectl get networkpolicy -n platform-mcp -o yaml   # ingress policy still present, now only selecting team-b
   # team-b's access to shared-k8sgpt-mcp still works
   kubectl exec -n team-b <agent-pod> -- curl -sS http://shared-k8sgpt-mcp.platform-mcp.svc.cluster.local:8089/mcp

7. Wrap-up talking point: this is the same composition + relationship-tracking model
   KubePlus already uses for SaaS instances, applied to agentic workloads — one annotation
   contract, chart-agnostic, giving a platform team per-tenant agent isolation with shared,
   governed access to common infrastructure (MCP servers, and later anything else a chart
   author wants to declare a dependency on).