From fdbd3873db3b4a6ea329b166acf95e724c2778ba Mon Sep 17 00:00:00 2001 From: Vincent Shen Date: Mon, 20 Apr 2026 00:57:41 -0700 Subject: [PATCH] Add AGENTS.md and contributor skill playbooks AGENTS.md documents the project layout, controller patterns, AIDE configuration quirks, Dockerfile topology, kubebuilder markers, test styles, and file-size guidance for the repo. Mirrors the top-level conventions doc pattern used by other agent-assisted projects (e.g. openai/codex). Three focused skill playbooks under .claude/skills/ cover the recurring FIO workflows an agent is likely to drive: - prow-logs: fetch OpenShift Prow CI artifacts (build logs, JUnit, gather-extra) for a failing presubmit job. - e2e-triage: how the operator is deployed during make e2e, five paths to validate a feature branch, live-state inspection commands, result-ConfigMap decode, and known failure shapes. - release-fio: the two-PR backport-plus-release dance for a z-stream (1.3.X), and the .tekton / Konflux setup for a new y-stream release branch (1.4.0). Documentation-only; no runtime code changes. --- .claude/skills/e2e-triage/SKILL.md | 169 ++++++++++++++++++++ .claude/skills/prow-logs/SKILL.md | 53 +++++++ .claude/skills/release-fio/SKILL.md | 237 ++++++++++++++++++++++++++++ AGENTS.md | 125 +++++++++++++++ 4 files changed, 584 insertions(+) create mode 100644 .claude/skills/e2e-triage/SKILL.md create mode 100644 .claude/skills/prow-logs/SKILL.md create mode 100644 .claude/skills/release-fio/SKILL.md create mode 100644 AGENTS.md diff --git a/.claude/skills/e2e-triage/SKILL.md b/.claude/skills/e2e-triage/SKILL.md new file mode 100644 index 000000000..dcf5e169e --- /dev/null +++ b/.claude/skills/e2e-triage/SKILL.md @@ -0,0 +1,169 @@ +--- +name: e2e-triage +description: Diagnose or run a file-integrity-operator e2e test — know how the operator is deployed during `make e2e`, scope a single test, point it at a feature-branch image, inspect operator/DaemonSet logs and result ConfigMaps. Use when the Prow `e2e-aws` / `e2e-bundle-aws` job fails, or when reproducing and validating against a real cluster. +--- + +# Triage or run a file-integrity-operator e2e test + +## How the operator is deployed during `make e2e` + +The test harness installs the operator itself unless you explicitly hand it a pre-installed bundle. Flow (driven by `Makefile` + `tests/framework/` + `tests/e2e/helpers.go`): + +1. **`make e2e-set-image`** writes the operator image pullspec into `config/manager/kustomization.yaml`. + - If `IMAGE_FROM_CI` is set → that wins (Prow uses this). + - Else → `$(IMG)` = `$(IMAGE_TAG_BASE):$(TAG)`, default `quay.io/file-integrity-operator/file-integrity-operator:latest`. +2. **`make prep-e2e`** kustomize-builds two manifests into `tests/_setup/`: + - `crd.yaml` (from `config/crd`) + - `deploy_rbac.yaml` (from `config/e2e`, which is `config/rbac` + `config/manager` — no namespace, no CRD) +3. **`go test ./tests/e2e`** runs with `-root=$(PROJECT_DIR) -globalMan=tests/_setup/crd.yaml -namespacedMan=tests/_setup/deploy_rbac.yaml -skipCleanupOnError=$(E2E_SKIP_CLEANUP_ON_ERROR) [--platform openshift|rosa]`. +4. Inside the test, `setupFileIntegrityOperatorCluster` (`tests/e2e/helpers.go:467`): + - rewrites the literal `openshift-file-integrity` to the actual test namespace in `deploy_rbac.yaml`, + - if `TEST_BUNDLE_INSTALL` **not** set → `ctx.InitializeClusterResources` applies the CRD + RBAC + operator `Deployment` from those manifests, + - if `TEST_BUNDLE_INSTALL` **is** set (any non-empty value) → assumes the operator is already installed (e.g. via OLM bundle) and skips creation, + - seeds the metrics-scrape RBAC (ClusterRole, ClusterRoleBinding, SA token Secret), + - waits for `deploy/file-integrity-operator` to become Available. + +## Environment variables that matter + +| Var | Purpose | Default | +|---|---|---| +| `IMG` | operator image pullspec used by `e2e-set-image` | `$(IMAGE_TAG_BASE):$(TAG)` | +| `IMAGE_FROM_CI` | overrides `IMG`; Prow sets this from its build | unset | +| `E2E_GO_TEST_FLAGS` | passed to `go test` | `-v -timeout 90m` | +| `E2E_SKIP_CLEANUP_ON_ERROR` | keep failing-test state for inspection | `true` | +| `TEST_OPERATOR_NAMESPACE` | namespace the operator deploys into | kubeconfig default | +| `TEST_WATCH_NAMESPACE` | namespace the operator watches | matches operator namespace | +| `TEST_BUNDLE_INSTALL` | skip operator install, assume bundle already on cluster | unset | + +## Validating a feature branch + +Four paths, fastest to slowest feedback: + +### 1. Targeted single test against an existing image + +Fastest loop when iterating on one scenario: + +```bash +# if IMG is already right (from make push): +E2E_GO_TEST_FLAGS="-v -timeout 20m -run TestFileIntegrityPriorityClassName" make e2e +``` + +### 2. Your fork's image from Quay (or any external registry) + +```bash +export IMAGE_REPO=quay.io/ +export TAG=$(git rev-parse --short HEAD) +make images # builds operator + bundle +make push # pushes to $IMAGE_REPO/file-integrity-operator{,-bundle}:$TAG +make e2e IMG=$IMAGE_REPO/file-integrity-operator:$TAG +``` + +### 3. OpenShift in-cluster registry (no external push) + +Handy on a personal cluster with no Quay credentials: + +```bash +make deploy-local +# under the hood: make install + make image-to-cluster (pushes to the cluster's +# internal registry as image-registry.openshift-image-registry.svc:5000/openshift/ +# file-integrity-operator:$TAG) + make deploy with that pullspec. +make e2e # test framework will see the already-deployed operator +``` + +Note: `deploy-local` patches `config/manager/deployment.yaml` in place and reverts it; if the revert fails, `git restore config/manager/deployment.yaml config/manager/kustomization.yaml`. + +### 4. Against an OLM-installed bundle + +When you've installed the operator through a catalog (mirrors what customers do): + +```bash +TEST_BUNDLE_INSTALL=1 \ + TEST_WATCH_NAMESPACE=openshift-file-integrity \ + TEST_OPERATOR_NAMESPACE=openshift-file-integrity \ + make e2e +``` + +### 5. Prow `e2e-aws` on the PR (what landing cares about) + +The authoritative signal for merge: + +```bash +gh pr comment -R openshift/file-integrity-operator -b "/test e2e-aws" +``` + +Prow will build the image from your branch HEAD and run the full suite. Logs via the `prow-logs` skill. + +## Scope a single test locally + +The suite is ~90 min. Scope with the go test `-run` flag (test names in `tests/e2e/e2e_test.go` all start with `TestFileIntegrity` or `TestMetrics` / `TestServiceMonitoring`): + +```bash +E2E_GO_TEST_FLAGS="-v -timeout 20m -run TestFileIntegrityConfigurationStatus" make e2e +``` + +Force cleanup on failure (the default `true` *skips* cleanup so state is inspectable): + +```bash +E2E_SKIP_CLEANUP_ON_ERROR=false make e2e +``` + +For ROSA: `make e2e-rosa` passes `--platform rosa` which skips MachineConfig-related operations and schemes. + +## Inspect live state + +All resources in `openshift-file-integrity` (or your `TEST_OPERATOR_NAMESPACE`): + +```bash +# Operator +oc -n openshift-file-integrity get deploy,pods -l name=file-integrity-operator +oc -n openshift-file-integrity logs deploy/file-integrity-operator + +# AIDE DaemonSets (one per FileIntegrity CR, plus short-lived reinit DS) +oc -n openshift-file-integrity get ds,pods -l app=aide-ds- +oc -n openshift-file-integrity logs ds/aide-ds- -c aide + +# CRs +oc get fileintegrities.fileintegrity.openshift.io -A +oc get fileintegritynodestatuses -A + +# Events +oc -n openshift-file-integrity get events --field-selector reason=FileIntegrityStatus +oc -n openshift-file-integrity get events --field-selector reason=NodeIntegrityStatus +``` + +## Read the failure log + +Failure details live in a result ConfigMap linked from the `FileIntegrityNodeStatus`: + +```bash +NS=openshift-file-integrity +oc get fileintegritynodestatus/ -n $NS -o yaml # find resultConfigMapName +oc get cm/ -n $NS -o jsonpath="{ .data.integritylog }" +``` + +If the ConfigMap carries annotation `file-integrity.openshift.io/compressed`, decode: + +```bash +oc get cm/ -n $NS -o jsonpath="{ .data.integritylog }" | base64 -d | gunzip +``` + +## Metrics sanity check + +```bash +oc run --rm -i --restart=Never \ + --image=registry.fedoraproject.org/fedora-minimal:latest \ + -n openshift-file-integrity metrics-test -- bash -c \ + 'curl -ks -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://metrics.openshift-file-integrity.svc:8585/metrics-fio' | grep file_integrity +``` + +## From Prow artifacts + +For PR-job failures, fetch operator + aide-ds pod logs from `artifacts/e2e-aws/test/artifacts/` and `gather-extra/artifacts/pods/`. Use the `prow-logs` skill to derive the URLs. + +## Known failure shapes + +- **Mount-propagation errors (CSI + multipath)** — expect `mountPropagation: HostToContainer` (#424, OCPBUGS-14947). +- **False positives after scaling a node** — MCO annotation files / kubelet CA; default config excludes them (#368, #413, #534). +- **Reinit stuck** — `holdoff` annotation race; per-node holdoff fixes it (#339). Check `file-integrity.openshift.io/node-holdoff-*` annotations on the FileIntegrity. +- **FIPS guard termination** — LD_PRELOAD MD5 guard working as intended (#660, OCPBUGS-56409). Exit code `64` = `MD5_GUARD_ERROR`. +- **Daemon starts before metrics secrets exist** — retry logic added in #821 (CMP-3757) and #845. diff --git a/.claude/skills/prow-logs/SKILL.md b/.claude/skills/prow-logs/SKILL.md new file mode 100644 index 000000000..c3f633ef0 --- /dev/null +++ b/.claude/skills/prow-logs/SKILL.md @@ -0,0 +1,53 @@ +--- +name: prow-logs +description: Fetch OpenShift Prow CI artifacts for a file-integrity-operator PR — build logs, JUnit XML, gather-extra cluster artifacts. Use when diagnosing a failing presubmit job on openshift/file-integrity-operator. +--- + +# Prow logs for file-integrity-operator + +## Find the build ID + +```bash +gh pr checks --repo openshift/file-integrity-operator +``` + +Links go to `https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_file-integrity-operator///`. The trailing integer is the build ID. + +UI alternative: `https://prow.ci.openshift.org/pr-history?org=openshift&repo=file-integrity-operator&pr=`. + +## Artifact URLs + +Base: + +``` +https://storage.googleapis.com/test-platform-results/pr-logs/pull/openshift_file-integrity-operator//// +``` + +Files worth fetching: + +| Path | Purpose | +|---|---| +| `build-log.txt` | test container stdout/stderr. Start here. | +| `finished.json`, `started.json` | pass/fail + timing | +| `artifacts///build-log.txt` | per-step logs for multi-step (e2e) jobs | +| `artifacts///artifacts/junit*.xml` | JUnit results | +| `artifacts//gather-extra/artifacts/pods/` | operator + DaemonSet pod logs (e2e only) | +| `artifacts//gather-extra/artifacts/events.json` | cluster events (e2e only) | + +Browse a full run tree: `https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_file-integrity-operator////` + +## Job names + +Pattern: `pull-ci-openshift-file-integrity-operator--` where `` is: + +Required: `unit`, `verify`, `go-build`, `images`, `ci-index-file-integrity-operator-bundle`, `e2e-aws` +Optional: `e2e-bundle-aws`, `e2e-bundle-aws-upgrade`, `e2e-rosa` + +`` is `master` or `release-4.XX` / `release-1.3`. Periodics: `periodic-ci-openshift-file-integrity-operator-master-nightly-4.XX-*`. + +## Rerun commands (PR comments) + +- `/retest` — all failed required jobs. +- `/retest-required` — only mandatory failures. +- `/test ` — one specific job, e.g. `/test e2e-aws`. +- `/ok-to-test` — gate for first-time contributors; org members only. diff --git a/.claude/skills/release-fio/SKILL.md b/.claude/skills/release-fio/SKILL.md new file mode 100644 index 000000000..121b7f7d2 --- /dev/null +++ b/.claude/skills/release-fio/SKILL.md @@ -0,0 +1,237 @@ +--- +name: release-fio +description: Cut a file-integrity-operator release — the two-PR backport + release dance for z-stream (1.3.X), or the new-branch + .tekton setup for y-stream (1.4.0). Use when the user asks to release, tag, cut a new FIO version, or create a new release branch. +--- + +# Release file-integrity-operator (z-stream 1.3.X) + +## What actually happens (from PRs #786, #793, #846, #849) + +A z-stream release is **two PRs** on `release-1.3`, not one: + +1. **Backport PR** — cherry-picks master fixes onto `release-1.3`. +2. **Release PR** — bumps version in a fixed set of files. + +Konflux handles downstream image/bundle rebuilds via `chore(deps): update file-integrity-operator-release-1-3 to ` PRs after merge. + +The `Makefile` has `prepare-release` / `push-release` / `release-images` targets, but **recent releases have diverged** from them (see "Makefile vs practice" below). Do not blindly run `make push-release`. + +## Preconditions + +- Maintainer with push access to `openshift/file-integrity-operator` and `quay.io/file-integrity-operator`. +- `IMAGE_REPO` and `TAG` are **unset** in the environment. +- All fixes intended for this release are already merged on `master` with `CMP-NNNN:` or `OCPBUGS-NNNNN:` titles (both satisfy `jira/valid-reference`). +- `VERSION=1.3.X` is set (semver, e.g. `1.3.9`). + +### Preflight — clear the queue on the target branch + +Before the backport and release PRs go out, flush both of these on the branch you're releasing from (`release-1.3` for a z-stream): + +1. **Refresh Go dependencies.** Land a `Update all Go dependencies to latest versions` PR (see `b309f11b6`, or `7ca340fa4` when a Go toolchain bump is bundled). Workflow: `go get -u ./... && go mod tidy && go mod vendor`, then `make test-unit && make verify`, commit and merge. Shipping a release on stale deps reopens CVE exposure. +2. **Merge every open Konflux / Tekton pipeline PR** on the target branch: + - `gh pr list --repo openshift/file-integrity-operator --base release-1.3 --state open` + - Merge pending `Update Konflux references`, `chore(deps): update file-integrity-operator-release-1-3 to `, and any `Re-add ... pipeline customization` follow-ups. + - Leaving them open means the release lands onto a branch whose pipeline config will race the next bot refresh, and FIO customizations (hermetic, arch list, prefetch, build-nudge) may silently drop during the release window. + +Only after both queues are empty (or an open item is explicitly deferred for a documented reason) should Phase 1 proceed. + +## Phase 1 — Backport PR + +Cherry-pick the master fixes that should ship. + +```bash +git fetch origin +git checkout release-1.3 +git pull --ff-only +git checkout -b backport-1.3.-fixes + +# For each merged master PR, cherry-pick its merge commit: +git cherry-pick -x -m 1 # -m 1 for merge commits only +# (for squash-merged commits, drop -m 1) + +make test-unit +make verify +git push -u origin backport-1.3.-fixes +``` + +Open the PR (body template from #846): + +```bash +gh pr create --repo openshift/file-integrity-operator \ + --base release-1.3 \ + --title "Backport fixes for 1.3. release" \ + --body "Cherry-picking the commit fixing these issues: +[CMP-NNNN](https://issues.redhat.com/browse/CMP-NNNN): (#) +[CMP-MMMM](https://issues.redhat.com/browse/CMP-MMMM): (#)" +``` + +Wait for merge before Phase 2. + +## Phase 2 — Release PR + +Only after the backport PR is merged. + +```bash +git checkout release-1.3 +git pull --ff-only +git checkout -b release-1.3. # NOTE: no "v" prefix (see divergence below) +``` + +Touch exactly these files (from the 1.3.8 diff, commit `99ba9364a`): + +| File | Change | +|---|---| +| `version/version.go` | `Version = "1.3."` | +| `version.Makefile` | `VERSION?=1.3.` | +| `build/Dockerfile.openshift` | `version=1.3.` LABEL | +| `bundle.openshift.Dockerfile` | `FIO_OLD_VERSION="1.3."`, `FIO_NEW_VERSION="1.3."` | +| `bundle/manifests/file-integrity-operator.clusterserviceversion.yaml` | `name: file-integrity-operator.v1.3.`, `olm.skipRange: '>=1.0.0 <1.3.'`, `version: 1.3.`, `replaces: file-integrity-operator.v1.3.` | +| `catalog/preamble.json` | entry `name` + `skipRange` | +| `config/manifests/bases/file-integrity-operator.clusterserviceversion.yaml` | `olm.skipRange` (may or may not need bumping — check current value; in 1.3.8 it lagged and was corrected in the release PR) | + +You can drive most of this via the Makefile (but staged, not pushed — see divergence): + +```bash +make update-skip-range VERSION=1.3. # rewrites the CSV/preamble skip ranges +make bundle VERSION=1.3. # regenerates bundle/manifests/ +# Then hand-edit version/version.go, version.Makefile, build/Dockerfile.openshift, +# bundle.openshift.Dockerfile (or use sed). +``` + +Commit as **one** commit: + +```bash +git add version/version.go version.Makefile build/Dockerfile.openshift \ + bundle.openshift.Dockerfile bundle/manifests catalog/preamble.json \ + config/manifests/bases +git commit -m "Release 1.3. + +Tag a new z-stream release to ." +git push -u origin release-1.3. +``` + +Open the PR: + +```bash +gh pr create --repo openshift/file-integrity-operator \ + --base release-1.3 \ + --title "Release 1.3." \ + --body "Tag a new z-stream release to ." +``` + +Merge gates: `approved` + `lgtm` + typically `qe-approved`. + +## Phase 3 — After merge + +- Tag the release: + + ```bash + git fetch origin + git checkout release-1.3 + git pull --ff-only + git tag v1.3. + git push origin v1.3. + ``` + +- `make release-images VERSION=1.3.` pushes versioned + `latest` tags to `quay.io/file-integrity-operator/file-integrity-operator{,-bundle,-catalog}`. Requires Quay write access. +- Konflux bot opens one or more `chore(deps): update file-integrity-operator-release-1-3 to ` PRs (e.g. #848, #850). Merge these; they wire the new SHA into downstream bundle/catalog builds. +- OSBS/Konflux does the downstream Red Hat image build via `.tekton/` pipelines and `build/Dockerfile.openshift`. + +## Makefile vs practice + +The Makefile documents a `make prepare-release / push-release / release-images` pipeline. Recent releases diverge: + +- `make push-release` commits as `Release v` and creates branch `release-v` (with `v`). Recent PRs use `Release 1.3.X` and `release-1.3.X` (no `v`). +- `make push-release` also merges to `ocp-1.0` and pushes it — this looks legacy; recent z-stream releases land via PR on `release-1.3`. +- `make prepare-release` stages `CHANGELOG.md`, but PRs #786 and #849 didn't update it. (Last CHANGELOG entry is 1.3.4.) + +Safe play: use the Makefile for `update-skip-range` and `bundle` (mechanical edits), but do the branch/commit/PR dance manually as above. + +## Watch-outs + +- **`config/manifests/bases/…` skipRange drift** — in #849 this file jumped from `<1.3.5-dev` to `<1.3.8`, meaning it had been missed in earlier releases. Always verify it matches the new version. +- **`bundle.openshift.Dockerfile` has dual ARGs** (`FIO_OLD_VERSION`, `FIO_NEW_VERSION`) — both must be bumped, and `FIO_OLD_VERSION` must equal the previous release. +- **`build/Dockerfile.openshift` version label** — easy to miss; fixed in #753 after it was left blank in a past release. +- **Don't retag** — if an image needs to change, bump to the next patch version. Released tags are immutable. + +## `.tekton/` during a z-stream release + +**Nothing.** Release PRs #786, #849 don't modify any `.tekton/*.yaml`. Konflux builds fire automatically on each push to `release-1.3` through the existing `file-integrity-operator-release-1-3-{pull-request,push}.yaml` + `file-integrity-operator-bundle-release-1-3-{pull-request,push}.yaml` pipelines. + +The operator push pipeline carries `build.appstudio.openshift.io/build-nudge-files: "bundle-hack/update_csv.go"` — after the operator image builds, Konflux re-runs `update_csv.go` against the bundle component and produces the `chore(deps): update file-integrity-operator-release-1-3 to ` follow-up PRs that bump the CSV image SHA automatically. **Don't remove that annotation; don't manually edit CSV image SHAs on release-1.3.** + +--- + +# Cutting a new release branch (y-stream, e.g. 1.4.0) + +This is the scenario where `.tekton/` changes are required. Procedure reconstructed from `ce4525e4a6` (branch init), `28c7d9d66` (rename), `36b22b2ba` (delete stale), `8bdbe36b8` (re-add customizations). + +## 0. Preflight on master + +Before branching, clear the same two queues on `master` that Phase 1 clears on `release-1.3` for z-streams: merge a fresh `Update all Go dependencies to latest versions` PR, and merge every open Konflux / Tekton pipeline PR. Cutting `release-1.4` off a master that still has pending pipeline refreshes inherits the staleness into the new branch and creates avoidable merge churn during the first weeks of the new stream. + +## 1. Branch off master + +```bash +git fetch origin +git checkout -b release-1.4 origin/master +``` + +## 2. Rename the four `.tekton/` files + +Master uses `-dev` as the Konflux component suffix. Copy and rename to the new branch suffix (dashes in filenames, dots only in `target_branch` CEL values): + +- `file-integrity-operator-dev-pull-request.yaml` → `file-integrity-operator-release-1-4-pull-request.yaml` +- `file-integrity-operator-dev-push.yaml` → `file-integrity-operator-release-1-4-push.yaml` +- `file-integrity-operator-bundle-dev-pull-request.yaml` → `file-integrity-operator-bundle-release-1-4-pull-request.yaml` +- `file-integrity-operator-bundle-dev-push.yaml` → `file-integrity-operator-bundle-release-1-4-push.yaml` + +## 3. Edit inside each file + +Replace every identifier that embeds the branch: + +| Field | From | To | +|---|---|---| +| `metadata.name` | `file-integrity-operator[-bundle]-dev-on-{pull-request,push}` | `...-release-1-4-on-{pull-request,push}` | +| `labels.appstudio.openshift.io/application` | `file-integrity-operator-dev` | `file-integrity-operator-release-1-4` | +| `labels.appstudio.openshift.io/component` | `file-integrity-operator[-bundle]-dev` | `file-integrity-operator[-bundle]-release-1-4` | +| `on-cel-expression` `target_branch ==` match | `"master"` | `"release-1.4"` (dots) | +| Bundle-path CEL regex fragment `-bundle-dev-.*\\.yaml` | `-dev-` | `-release-1-4-` | +| `spec.params.output-image` | `.../file-integrity-operator[-bundle]-dev:...` | `.../file-integrity-operator[-bundle]-release-1-4:...` | +| `spec.taskRunTemplate.serviceAccountName` | `build-pipeline-file-integrity-operator[-bundle]-dev` | `build-pipeline-file-integrity-operator[-bundle]-release-1-4` | + +## 4. Re-add FIO-specific customizations + +Konflux stock templates drop these. They must be explicitly re-added every time the bot refreshes a template (see `8bdbe36b8`): + +- `spec.params.build-platforms: [linux/x86_64, linux/ppc64le, linux/s390x]` — **no arm64** (removed in `c24ef5f54`). +- `spec.params.hermetic: "true"` +- `spec.params.build-source-image: "true"` +- `spec.params.prefetch-input: '[{"type": "rpm", "path": "konflux"}, {"type": "gomod", "path": "."}]'` +- On the `prefetch-dependencies` task: param `dev-package-managers: "true"`. +- On the operator push pipeline: the `ADDITIONAL_TAGS: ['{{ target_branch }}']` stanza in the `push-dockerfile` finally block (from PR #795). +- On the operator push pipeline: annotation `build.appstudio.openshift.io/build-nudge-files: "bundle-hack/update_csv.go"`. + +## 5. Delete stale `.tekton/*.yaml` + +If the new branch carries any `file-integrity-operator-*-{dev,master,release-1-3}-*.yaml`, delete it — those will cross-fire against the wrong Konflux application. Commit `36b22b2ba` did exactly this on release-1.3. + +## 6. Konflux admin setup (out-of-band) + +- Provision Konflux application `file-integrity-operator-release-1-4` with both components. +- Create `build-pipeline-file-integrity-operator[-bundle]-release-1-4` service accounts. +- Configure the Pipelines-as-Code webhook for the new branch. + +These happen in the Konflux UI / tenant admin flow (`ocp-isc-tenant`), not in this repo. Coordinate with whoever owns the Konflux tenant before pushing the branch. + +## 7. Verify + +Push a trivial commit to `release-1.4` and confirm both operator and bundle pipelines fire, producing images at `quay.io/redhat-user-workloads/ocp-isc-tenant/file-integrity-operator[-bundle]-release-1-4:*`. + +--- + +# Ongoing `.tekton/` maintenance (any branch) + +Konflux bot raises periodic `Update Konflux references` and `chore(deps): …` PRs that refresh pipeline template SHAs. These refreshes routinely drop the customizations from step 4 above. Commits titled "Re-add operator pipeline customization" / "Readd bundle pipeline customizations" show this is a recurring chore, not a one-time setup. On every bot PR, diff against the prior state and re-add anything dropped (usually in a fast-follow commit on the same PR). + +**Trigger separation:** the operator and bundle pipelines carry inverted CEL expressions on the bundle paths `^bundle/`, `^bundle-hack/`, `^bundle\.openshift\.Dockerfile`, `^\.tekton/file-integrity-operator-bundle-*-.*\.yaml`. Operator pipeline runs when those paths **didn't** change; bundle pipeline runs **only** when they did. Adding a new bundle-impacting path means updating both CEL lists. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..382f0446d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,125 @@ +# file-integrity-operator + +OpenShift Operator that runs AIDE file-integrity scans on cluster nodes via a privileged DaemonSet. Module path: `github.com/openshift/file-integrity-operator`. Dependencies are vendored (`-mod=vendor`). + +Code layout: + +- `pkg/apis/fileintegrity/v1alpha1/` — CRD types (`FileIntegrity`, `FileIntegrityNodeStatus`). CR group: `fileintegrity.openshift.io/v1alpha1`. +- `pkg/controller/{fileintegrity,configmap,node,status,metrics}/` — one reconciler per package. +- `pkg/common/` — shared constants, helpers. `constants.go` holds every annotation/label key and AIDE error code used across the codebase. +- `cmd/manager/` — a single Cobra app with two subcommands. `operator.go` registers `OperatorCmd` (controller-runtime manager); `daemon.go` registers `DaemonCmd` (runs per-node in the DaemonSet, drives the AIDE loop and the log-collector loop via `loops.go` + `logcollector_util.go`). Only these two subcommands are wired in `main.go`; `build/bin/entrypoint` is a trivial `exec ${OPERATOR} $@` wrapper. +- `tests/e2e/` — e2e tests; framework in `tests/framework/`. +- `bundle-hack/` — **downstream build-time tooling, not runtime.** `update_csv.go` rewrites the CSV during the Konflux build (used for image-SHA pinning). `update_bundle_annotations.sh` writes OCP version / channel into `bundle/metadata/annotations.yaml`. `rpms.lock.yaml` pins RPMs for Konflux hermetic builds. Invoked from `.tekton/` pipelines, not from `make`. + +## Sources of truth + +Read from these files rather than duplicating their contents anywhere: + +- `go.mod` — Go version and direct dependencies. +- `Makefile`, `version.Makefile` — build targets, `VERSION`, tool versions (operator-sdk, kustomize). +- `PROJECT` — kubebuilder scaffold (resources, API groups). +- `OWNERS` — approvers and reviewers. +- `pkg/common/constants.go` — all annotation/label keys and AIDE error codes. Never hardcode strings like `"file-integrity.openshift.io/re-init"`; use the constant. + +## Before finishing any change + +- `make test-unit` (fmt + vet + unit tests) and `make verify` (vet + `gosec` at severity/confidence ≥ medium). These mirror the required Prow jobs `unit` and `verify`. +- Edited `pkg/apis/fileintegrity/v1alpha1/*`? Run `make manifests` (CRDs) and `make generate` (DeepCopy). +- Edited CSV, RBAC, or anything that flows into the OLM bundle? Run `make bundle`, then `hack/tree-status` — drift fails the Prow `verify` job. `make bundle` also rewrites `config/manager/kustomization.yaml` as a side effect; the Makefile restores it via `git restore`, but double-check it's clean before committing. To pin the CSV image: `make bundle IMG=quay.io/file-integrity-operator/file-integrity-operator:`. +- Dependency changes: `go mod tidy && go mod vendor`. Never edit `vendor/` by hand. + +## Generated / don't-hand-edit + +Always regenerate, never edit directly: + +- `bundle/manifests/` and `bundle/metadata/annotations.yaml` — from `make bundle`. +- `config/crd/bases/fileintegrity.openshift.io_*.yaml` — from `make manifests`. +- `config/rbac/role.yaml` — from `make manifests` (driven by `+kubebuilder:rbac:` markers). +- `pkg/apis/fileintegrity/v1alpha1/zz_generated_*.go` — from `make generate`. +- `vendor/` — from `go mod vendor`. + +## Types and kubebuilder markers + +- **Two CRs, deliberately split.** `FileIntegrity` (cluster-scoped config, `.Status.Phase` only) lives alongside `FileIntegrityNodeStatus` (per-node scan results, owns `Results[]` + `LastResult`). Don't stuff per-node status into `FileIntegrity.Status`; use a new `FileIntegrityNodeStatus` or extend the existing scan-result fields. +- **Phase/condition vocabulary is fixed.** `FileIntegrityStatusPhase` is one of `Initializing | Active | Pending | Error`. `FileIntegrityNodeCondition` is `Succeeded | Failed | Errored`. Don't invent new values; consumers (metrics, events, alerts) key on these exact strings. +- **RBAC markers are centralized** in `pkg/controller/fileintegrity/setup.go`, above the `FileIntegrityReconciler.Reconcile` comment block. All controller RBAC needs live there — do not scatter `+kubebuilder:rbac:` comments across individual reconcilers, even though tooling would accept it. Adding a new permission means editing that single file, then `make manifests`. +- **CRD markers** (`+kubebuilder:validation:…`, `+kubebuilder:default=…`, `+kubebuilder:printcolumn:…`, `+kubebuilder:subresource:status`) sit on type fields in `pkg/apis/fileintegrity/v1alpha1/fileintegrity_types.go` and drive `config/crd/bases/` via `make manifests`. Defaults belong in the marker, not in controller logic — see existing `+kubebuilder:default=900` on `GracePeriod` and the structured `Tolerations` default for reference. + +## Controller patterns + +- **Reconciler skeleton is split across two files per package.** `setup.go` holds the struct definition, the `Reconcile` interface-matching wrapper, and `SetupWithManager`; the sibling `*_controller.go` file holds the real reconcile logic (called via a helper like `r.ConfigMapReconcile(req)` or `r.FileIntegrityControllerReconcile(req)`). Follow this split for new reconcilers. The `_ = ctrlLog.FromContext(ctx)` line in every `setup.go` is kubebuilder scaffold noise; don't propagate it into new logic — the codebase logs through package-scoped `logf.Log.WithName(...)` vars instead. +- **Three-function wire-up per controller package.** Every controller exposes an exported `AddController(mgr, *metrics.Metrics) error` that `cmd/manager/operator.go` calls, plus two private helpers: `newReconciler` builds the struct with `Client`, `Scheme`, `Recorder`, `Metrics`; `addController` wires `NewControllerManagedBy(mgr).Named("-controller").For(&PrimaryType).Watches(…).Complete(r)`. Follow this shape; don't collapse the three functions. +- **Secondary-resource watches use a mapper in a sibling file.** When a reconciler needs to react to events on a different type (FileIntegrity watching ConfigMap, for example), put the translator in `_mapper.go` implementing `handler.MapFunc` (see `fileintegrity_cm_mapper.go` for the template). Mapper lists the primary CRs and returns `[]reconcile.Request` for the ones affected. +- **Reconcile returns use `reconcile.Result`** (from `sigs.k8s.io/controller-runtime/pkg/reconcile`), not `ctrl.Result`, for consistency with existing code. Canonical forms: `return reconcile.Result{}, nil` (done), `return reconcile.Result{}, err` (controller-runtime requeues with backoff), `return reconcile.Result{Requeue: true}, nil`, `return reconcile.Result{RequeueAfter: }, nil`. +- **Event recorder is per-reconciler.** Each reconciler struct holds `Recorder record.EventRecorder`, initialized at setup via `mgr.GetEventRecorderFor("")` (see `fileintegrity_controller.go:52`, `configmap_controller.go:43`, `status_controller.go:33`). Emit with `r.Recorder.Eventf(obj, eventType, reason, format, args…)` directly — there is no `pkg/common` wrapper. Reason strings are short PascalCase nouns like `FileIntegrityStatus`, `NodeIntegrityStatus`, `PriorityClass`; don't invent per-case reasons. +- **Metrics go through a shared `*metrics.Metrics` struct** attached to reconcilers; increment via named methods like `r.metrics.IncFileIntegrityPhaseActive()`. Register new metrics in `pkg/controller/metrics/metrics.go` with names constructed from the `metricNamespace = "file_integrity_operator"` prefix; they expose on `/metrics-fio` (port `8585`). The counterfeiter-generated `metricsfakes/fake_impl.go` is committed but its `go:generate` line is commented out — if you change the `impl` interface, regenerate by hand with `go run github.com/maxbrunsfeld/counterfeiter/v6 -generate`. +- **Status updates are single-shot** — `r.client.Status().Update(ctx, deepCopy)` without `RetryOnConflict` wrapping (see `status_controller.go:172`). If you need conflict handling for a new hot-path update, introduce it deliberately. + +## Operator ↔ daemon runtime architecture + +The same binary runs in two modes (cobra subcommands `operator` and `daemon`). They communicate through three deliberately narrow channels — understand these before adding a new signal: + +- **Kubernetes CRs.** The operator owns `FileIntegrity` state. The daemon (running per-node in a privileged DaemonSet) reads the FI it belongs to via a dedicated dynamic client in `cmd/manager/daemon.go` — it is **not** a controller-runtime reconciler. +- **Files on the host filesystem.** The operator spawns a **separate reinit DaemonSet** (`reinitAideDaemonset` in `fileintegrity_controller.go`, distinct from the main `aideDaemonset` — two DaemonSets, not one) whose only job is to drop `/hostroot/run/aide.reinit`. The daemon's `reinitLoop` polls for that file. If you change the trigger path, update both the operator and the daemon. +- **Result ConfigMaps.** The daemon's log-collector writes scan results into per-node ConfigMaps in the operator namespace; the `ReconcileConfigMap` reconciler consumes them and creates/updates `FileIntegrityNodeStatus` objects. Results >1MB are gzip-compressed and base64-encoded with a `file-integrity.openshift.io/compressed` annotation (see `pkg/common/constants.go`). + +The daemon is cooperating goroutines (`aideLoop`, `reinitLoop`, `holdOffLoop`, `integrityInstanceLoop`, `logCollectorMainLoop` in `cmd/manager/loops.go` + `logcollector_util.go`) coordinated via error channels and a `sync.WaitGroup`. A new background task on the daemon side means a new loop registered in `daemonMainLoop`. + +User-provided AIDE configs go through `prepareAideConf` in `pkg/controller/fileintegrity/config.go`, which rewrites `database=`, `database_out=`, `report_url=file:`, `@@define DBDIR`, `@@define LOGDIR`, and prepends `/hostroot` to any absolute path or exclusion (`!/…`). If you add a new path-aware AIDE directive to the defaults, handle it here too. + +## Tests + +Three styles coexist — match the surrounding files when adding new tests. + +- **`cmd/manager/*_test.go` — Ginkgo / Gomega BDD** (`Describe`, `Context`, `When`, `It`, `Expect`). `manager_suite_test.go` is the Ginkgo entry point. Keep new operator-startup / PrometheusRule / webhook-plumbing tests here. +- **`pkg/**/*_test.go` — plain `testing.T`**, optionally with `stretchr/testify/require`. Counterfeiter fakes under `pkg/controller/metrics/metricsfakes/` are consumed by `metrics_test.go` (re-generate by hand if you change the `impl` interface — `go:generate` is intentionally commented out). +- **`tests/e2e/` — plain `testing.T` against a real cluster** via the custom wrapper in `tests/framework/`. Entry point is `setupTest(t)` returning `(*framework.Framework, *framework.Context, namespace)`. Reuse the `waitFor*` / `assert*` / `retryDefault` helpers already in `tests/e2e/helpers.go` — don't re-roll polling helpers (there are 98 of them; pick or extend one). + +Although `github.com/onsi/ginkgo` and `gomega` are present transitively, **don't introduce Ginkgo into `pkg/` or `tests/e2e/`** — only `cmd/manager/` uses it. + +## File size and placement + +Target Go source files under ~500 lines, tests excluded. A file past ~800 lines is a signal that the next feature belongs in a new module, not another function in the same file. Prefer new focused packages over growing the grab-bag packages. + +These files are already bloated — **don't extend them unless you're specifically refactoring them smaller.** Put new functionality in a sibling file or package: + +- `pkg/controller/fileintegrity/fileintegrity_controller.go` (~1080 LoC) — main reconciler. Split new feature logic into a sibling file (e.g. `fileintegrity_reinit.go`) or a sub-package. +- `pkg/common/util.go` (~570 LoC) — helper grab-bag. Treat `pkg/common` like a constrained library: resist adding to it. New helpers that naturally group (AIDE parsing, daemon helpers, node selection) belong in a focused package under `pkg/` or `pkg/common//`. +- `tests/e2e/helpers.go` (~2550 LoC) — e2e helper dumping ground. Split new helpers into topical files (`helpers_reinit.go`, `helpers_metrics.go`, `helpers_config.go`) rather than appending. +- `tests/e2e/e2e_test.go` (~1310 LoC) — main e2e test file. Group related new tests in a focused `*_test.go` file. + +Do not create small helper methods referenced only once; inline them. + +## AIDE configuration + +- Default config is `pkg/controller/fileintegrity/config_defaults.go`. At runtime the operator rewrites `database`, `database_out`, `report_url`, `DBDIR`, and `LOGDIR` to pod-appropriate paths. +- **Two AIDE config variants coexist for forward-compatibility.** The default vars (`DefaultAideConfigCommonStart`, `…End`) target AIDE 0.16 — **this is what production and all shipping downstream images run.** The `*018` vars target AIDE 0.18 and are gated on `AIDE_VERSION=0.18`; they are preparatory only and not in production use yet. If you change the default config, update both variants together so the 0.18 path doesn't silently diverge. +- `mhash` digests are not supported by the AIDE container; use the default `CONTENT_EX` group for digests. +- FIPS is enforced by an LD_PRELOAD MD5 guard (`build/guard/libaide_md5_guard.so`, installed at `/opt/libaide_md5_guard.so` — see `pkg/common/util.go`). It is built only when `libgcrypt-devel` is installed (`HAS_LIBCRYPT_DEV` gate in the Makefile). The guard hard-fails if AIDE tries to use MD5; exit code `64` (`MD5_GUARD_ERROR`) indicates a guard-triggered termination. +- Host filesystem is mounted under `/hostroot/` inside the daemon container. AIDE DB/log live at `/hostroot/etc/kubernetes/aide.db.gz{,.new}` and `/hostroot/etc/kubernetes/aide.log{,.new}`; the reinit trigger file is `/hostroot/run/aide.reinit`. + +## Four Dockerfiles — don't confuse them + +- `build/Dockerfile` — upstream / local dev (`golang` + `fedora-minimal:37` pinning AIDE 0.16). Entry point `/usr/local/bin/entrypoint`. +- `Dockerfile.ci` — Prow CI (`registry.ci.openshift.org/openshift/release:rhel-9-release-golang-*-openshift-*`) + AIDE 0.16. Used by the `images` presubmit. +- `Dockerfile.AIDE0.18.ci` — **preparatory-only** CI variant for eventual AIDE 0.18 support (not shipped; production and downstream are still 0.16). Installs unpinned `aide`, asserts the version is 0.18, sets `AIDE_VERSION=0.18`. Keep it building so the 0.18 path doesn't rot, but don't rely on it downstream. +- `build/Dockerfile.openshift` — downstream Konflux/OSBS. Uses `brew.registry.redhat.io/rh-osbs/openshift-golang-builder`, sets `BUILD_FLAGS=-tags strictfipsruntime`, requires `libgcrypt-devel`, carries Red Hat container labels. The `version=` LABEL is pinned manually — verify it on every release; stale values have shipped before. + +## Dev tools are vendored via `tools.go` + +`tools.go` at repo root uses the standard `//go:build tools` pattern to keep `gosec` and `controller-gen` reachable through `go mod vendor`. When a new dev-only binary is needed (linter, codegen tool), add a blank import here and run `go mod tidy && go mod vendor` so the build can resolve it from `vendor/` without a separate install step. + +## PR and commit title conventions + +- `CMP-NNNN: ` is the current convention and covers **both bug fixes and features** — FIO bugs live in the CMP Jira project now. +- `OCPBUGS-NNNNN: ` is legacy but still accepted for OCP-wide bug tracker items. +- Plain-English titles are fine for changes without a Jira. +- PR body links the Jira (`https://issues.redhat.com/browse/`). Both prefixes satisfy the `jira/valid-reference` Prow label. + +## Skills for procedural work + +`.claude/skills/` — invoke these instead of rederiving the procedure: + +- `prow-logs` — fetch CI artifacts (build logs, JUnit, gather-extra) for a failing Prow presubmit. +- `e2e-triage` — diagnose a failing e2e test; scope a single test; inspect pods/CRs/result ConfigMaps. +- `release-fio` — cut a z-stream 1.3.X release. The Makefile's release targets have drifted from practice; follow the skill, not `make push-release`.