Add evals#47
Merged
Merged
Conversation
Tier 1 (offline): 18 scenarios covering explain, apply --skip-apply,
initdata create/dump/validate, and shell completion — no cluster needed.
Tier 2 (cluster): api-reachable, apply-creates-object (skipped without
kata-cc RuntimeClass), and kbs-detect-or-skip (skipped without Trustee).
Run with: make eval-offline (Tier 1 only)
make eval (both tiers)
Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Updates apply-creates-object to check for and use kata-qemu-coco-dev instead of kata-cc, matching the CoCo dev Helm chart deployment. Install with: helm install coco oci://ghcr.io/confidential-containers/charts/confidential-containers --namespace coco-system --create-namespace Uninstall: helm uninstall coco --namespace coco-system && kubectl delete namespace coco-system Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds an eval/ black-box evaluation suite for kubectl-coco, plus Makefile targets to run Tier 1 (offline) and Tier 2 (cluster) scorecards. This fits into the repo as an end-to-end CLI workflow verification layer that complements existing cmd/... and integration_test/... tests.
Changes:
- Add
make eval-offlineandmake evaltargets to run the new eval test suite. - Introduce Tier 1 offline evals covering
explain,apply --skip-apply,initdata, and shell completion. - Introduce Tier 2 cluster evals that exercise real cluster workflows (namespace setup, init/kbs/apply flows) and print a scorecard.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| Makefile | Adds eval / eval-offline targets to execute the new eval suite. |
| eval/README.md | Documents eval purpose, how to run tiers, and what each scenario checks. |
| eval/helpers_test.go | Adds shared helpers: subprocess runner, fixtures copying, scorecard tracking/printing. |
| eval/tier1_offline_test.go | Adds Tier 1 offline black-box scenarios (no cluster required). |
| eval/tier2_cluster_test.go | Adds Tier 2 cluster end-to-end workflow scenarios (skips when no cluster). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+871
to
+878
| func createEvalNamespace(t *testing.T) { | ||
| t.Helper() | ||
| exec.Command("kubectl", "create", "namespace", evalNamespace).Run() //nolint:errcheck | ||
| t.Cleanup(func() { | ||
| exec.Command("kubectl", "delete", "namespace", evalNamespace, | ||
| "--ignore-not-found=true", "--timeout=60s").Run() //nolint:errcheck | ||
| }) | ||
| } |
Comment on lines
+883
to
+900
| func backupCertDir(t *testing.T) { | ||
| t.Helper() | ||
| home, err := os.UserHomeDir() | ||
| if err != nil { | ||
| t.Fatalf("UserHomeDir: %v", err) | ||
| } | ||
| certDir := filepath.Join(home, ".kube", "coco-sidecar") | ||
|
|
||
| // Snapshot existing contents before any test writes. | ||
| snapshot, snapshotErr := snapshotDir(certDir) | ||
|
|
||
| t.Cleanup(func() { | ||
| os.RemoveAll(certDir) //nolint:errcheck | ||
| if snapshotErr == nil && len(snapshot) > 0 { | ||
| restoreDir(certDir, snapshot) | ||
| } | ||
| }) | ||
| } |
Comment on lines
+935
to
+950
| func backupCocoConfig(t *testing.T) { | ||
| t.Helper() | ||
| home, err := os.UserHomeDir() | ||
| if err != nil { | ||
| t.Fatalf("UserHomeDir: %v", err) | ||
| } | ||
| cfgPath := filepath.Join(home, ".kube", "coco-config.toml") | ||
| original, readErr := os.ReadFile(cfgPath) | ||
| t.Cleanup(func() { | ||
| if readErr == nil { | ||
| os.WriteFile(cfgPath, original, 0o600) //nolint:errcheck | ||
| } else { | ||
| os.Remove(cfgPath) //nolint:errcheck | ||
| } | ||
| }) | ||
| } |
Comment on lines
+90
to
+94
| runBin(t, "apply", "-f", src, "--config", cfg, "--skip-apply", "--convert-secrets=false") | ||
| out, err := os.ReadFile(cocoOutput(src)) | ||
| if err != nil { | ||
| t.Fatalf("output file missing: %v", err) | ||
| } |
Replace the skip-if-no-Trustee guard with proper workflow scenarios: - cluster/kbs-start: runs `kubectl coco kbs start` and waits for Trustee - cluster/kbs-populate: uploads a test resource via the KBS HTTP API - cluster/apply-transforms-with-secrets: applies a pod with a K8s secret using --skip-apply, verifies the -coco.yaml has sealed secret refs - cluster/apply-creates-pod: creates a pod with kata-qemu-coco-dev (skipped unless the RuntimeClass is present) Also fix two assertion bugs: check for -sealed suffix (not 'sealed.' which never appears in the pod manifest), and embed namespace in the manifest so kubectl apply targets coco-eval not the context default. Also back up and restore ~/.kube/coco-config.toml so kbs start does not permanently alter the developer's config. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Tests that `kubectl coco init` creates ~/.kube/coco-config.toml and deploys Trustee to the eval namespace. The subsequent kbs-start test now exercises detection of an already-deployed Trustee rather than initial deployment. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Verifies that --init-container injects an initContainers section into the transformed manifest. Uses --skip-apply so no kata RuntimeClass is required; the test validates the transformation output only. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Verifies that apply works on a Deployment (not just Pod) by checking the Deployment object lands in the cluster with the expected name. Skipped when kata-qemu-coco-dev RuntimeClass is absent. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Exercises the --from-k8s-secret input mode of kbs populate, which is a separate code path from --path/--resource-file. Creates a real K8s secret then uploads all its keys to KBS via the HTTP API. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
After kbs-populate uploads a resource, kubectl exec into the Trustee pod to confirm the file actually exists in the KBS on-disk repository. This catches cases where populate exits 0 but the write silently failed. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Verifies --sidecar injects the coco-secure-access container into the transformed manifest. Uses --sidecar-skip-auto-sans to avoid needing a Service object; --skip-apply keeps this cluster-state independent. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Tests the end-to-end pipe: initdata create → dump (base64+gzip) → validate (reads encoded blob from stdin). Uses the config written by init/kbs-start so the initdata embeds the real in-cluster KBS URL. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
init now runs with --enable-sidecar so the client CA is present when apply-sidecar needs to sign the server certificate. apply-sidecar passes --sidecar-image (required when config has no image set) and an explicit --sidecar-san-dns so the cert generation succeeds without a live Service for auto-detection. backupCertDir helper added to restore ~/.kube/coco-sidecar/ state after the eval so init --enable-sidecar does not permanently alter the system. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
…ions Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Tests the volumes[].secret.secretName code path, which is distinct from env secretKeyRef conversion. Creates a K8s secret, applies a pod with a volume mount referencing it, then verifies: - -coco.yaml has the -sealed suffix on the volume secretName - trustee-secrets file is generated for the user to populate KBS - kbs populate -f uploads the key to KBS - kubectl exec confirms the key exists in the KBS repository Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Tests the 'I already have a KBS' flow: init with --skip-trustee-deploy and an explicit --trustee-url writes the config without touching the cluster. Verifies: - Command exits 0 - Config contains the provided KBS URL - Pod count in the eval namespace is unchanged (no new deployment) Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Tests cococtl apply on a two-document YAML (Pod + Service). Key behavioral assertion: cococtl extracts the primary workload (Pod), transforms it with cc_init_data and runtimeClassName, and saves only the Pod to -coco.yaml. The Service document is NOT applied — users apply it separately. Verifies: - -coco.yaml contains cc_init_data on the Pod - -coco.yaml does NOT contain the Service document - With kata RuntimeClass: Pod is created in the cluster, Service absent Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Runs cococtl apply twice on the same manifest and verifies both invocations exit 0. Two assertions: 1. --skip-apply (always): second run produces byte-identical -coco.yaml confirming sealed-secret token and initdata generation are stable. 2. Full cluster apply (skipped without kata): two consecutive apply calls both exit 0 and leave the pod object in the cluster (kubectl apply is idempotent; sealed-secret creation via kubectl apply is too). Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Tests `kbs start --mode external --url <url>` which registers a pre-existing KBS without any cluster interaction. Verifies: - Command exits 0 - Config contains the registered URL - Pod count in the namespace is unchanged (no deployment occurred) Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Verifies cococtl apply works on a StatefulSet with correct annotation placement. The critical invariant: cc_init_data must appear inside spec.template.metadata.annotations, NOT in the top-level StatefulSet metadata.annotations. The test splits the -coco.yaml on 'template:' and checks the annotation is absent before and present after. With kata RuntimeClass: StatefulSet object lands in the cluster and spec.template.spec.runtimeClassName is set correctly. Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.