CatalogSource pods should set openshift.io/required-scc annotation to prevent SCC preemption on OpenShift HCP

[TGPSKI]

Summary

On OpenShift HCP (Hosted Control Plane) clusters, CatalogSource pods can be assigned a customer-installed SCC instead of restricted-v2, causing the pod to run with an unexpected UID and crash with permission errors on cache directories.

This is actively causing production CrashLoopBackOff on ROSA HCP clusters.

Problem

OpenShift's SCC admission selects the "most restrictive" SCC from the candidate set of the creating user. On HCP, the entity that creates catalog pods on the data plane appears to have broader SCC access than the local catalog-operator ServiceAccount on classic OpenShift — evidenced by the security.openshift.io/validated-scc-subject-type: user annotation on affected pods. This means custom SCCs installed on the cluster can enter the candidate set and win admission.

A custom SCC with runAsUser.type: MustRunAs (single fixed UID) scores as more restrictive than restricted-v2's MustRunAsRange. When such an SCC wins, it mutates the pod to a UID that differs from the image's expected UID (1001 per USER directive in the image Dockerfile), resulting in permission errors on cache directories:

open /tmp/cache/pogreb.v1/db/lock: permission denied

This does not reproduce on classic OpenShift — the same custom SCC installed on classic causes restricted-v2 to still win, consistent with the local catalog-operator SA not having access to customer-installed SCCs.

Note: --set-workload-user-id=false is already set in the OpenShift OLM deployment, so OLM correctly does not inject a runAsUser. The UID mutation comes entirely from SCC admission selecting the custom SCC.

Expected Behavior

CatalogSource pods should consistently run under restricted-v2 on OpenShift, regardless of what custom SCCs are installed on the cluster.

Proposed Fix

OpenShift enhancement openshift/enhancements#1391 (AUTH-372, merged July 2023) introduced the openshift.io/required-scc pod annotation, which pins SCC selection to a specific SCC and fails admission hard rather than silently selecting a scored winner.

The fix should be applied in staging/operator-lifecycle-manager/pkg/controller/registry/reconciler/reconciler.go, in the Pod() function, within the existing Restricted security context branch — immediately after the addSecurityContext call:

if securityContextConfig == operatorsv1alpha1.Restricted {
    addSecurityContext(pod, runAsUser)
    // Pin SCC selection to restricted-v2 to prevent custom SCCs from preempting
    // this pod on OpenShift HCP, where the creating user has broader SCC access
    // than the local catalog-operator SA.
    if _, alreadySet := podAnnotations["openshift.io/required-scc"]; !alreadySet {
        podAnnotations["openshift.io/required-scc"] = "restricted-v2"
    }
}

Why this gate is correct: The Restricted branch fires when the namespace has PSA label pod-security.kubernetes.io/enforce: restricted or the user explicitly sets grpcPodConfig.securityContextConfig: restricted. On non-OpenShift clusters, the openshift.io/required-scc annotation is silently ignored (no SCC admission controller exists), making this a no-op outside OpenShift. The !alreadySet check respects any user-provided override passed through via CatalogSource annotations.

Additional Context

• On HCP, pod annotation security.openshift.io/validated-scc-subject-type: user indicates SCC admission ran against the creating user, not the pod's ServiceAccount
• Cross-platform reproduction confirms the access model differs: identical SCC definition on classic → restricted-v2 still wins
• This is an HCP-specific issue — classic clusters are not affected
• The openshift.io/required-scc annotation is GA since OCP 4.14+

Scope Note

This fix addresses CatalogSource pods specifically. Whether CSV-derived operator pods are also vulnerable to the same preemption on HCP (i.e., whether they are also created by the elevated HCP identity rather than the local olm-operator SA) is a separate investigation that could expand scope.