EVERYTHING Feat/proposal for e2e and user demo refactoring - #5
Open
sk31337 wants to merge 115 commits into
Open
Conversation
… setup in the test environment
…l in helm-simple examples
… maintainability
Stage 2 complete — validation green
Refactor delivered:
- test/e2e/setup/local.sh — orchestrator that runs components/*.sh in lexical order
- test/e2e/setup/components/00_prereqs.sh … 60_kro.sh — 7 idempotent component scripts, each with set -euo pipefail
- test/e2e/setup/manifests/{image-registries,rbac}.yaml — moved out of hacks/ (git mv, history preserved)
- test/e2e/hacks/setup.sh — shrunk from 157 → 11 lines, now a thin shim that execs setup/local.sh so the existing Taskfile target works unchanged
- 10_registry.sh includes the curl 127.0.0.1:5000/v2/ health check the user authorized, with an explicit AirPlay-collision hint on failure
Validation (per the user's chosen path: full teardown + setup + e2e):
- kind delete cluster --name kind → clean state
- task test/e2e/setup/local → all 7 components ran clean, kind cluster up, all controllers Ready
- task test/e2e → 12 Passed | 0 Failed | 0 Pending | 0 Skipped in 309.25s, plus all unit tests (TestSubstituteVars, TestLoadScenarioMinimal, TestWalkScenarios, etc.) PASS
… configurations Stage 3 done. Summary: - Migrated 9 examples to examples/<family>/<name>/ (helm, kustomize, k8s-manifest) - Authored 9 declarative e2e.yaml files - Implemented runScenario (deploy/wait/assert/hooks/cleanup) + new e2e_scenarios_test.go Ginkgo wiring - Added legacy_test.go shim for unmigrated specs (applyset, credentials) - Deleted e2e_examples_test.go - go vet clean, runner unit tests pass - Fresh-cluster e2e: 12/12 SUCCESS in 322s (one prior fresh-cluster run had a transient ArgoCD cold-start flake on the first ArgoCD-bearing scenario; saved as project memory)
Update DESIGN.md and AUTHORING.md to reflect the locked decision (Q5b) that within examples/helm/, scenarios are split a second level by delivery tool: helm/fluxcd/<name>/ for Flux-only and helm/argocd/<name>/ for ArgoCD-only. Each scenario uses exactly one tool. Parity demos belong under test/e2e/scenarios/helm/parity/.
Duplicate the 6 helm/* scenarios into two parallel subtrees: helm/fluxcd/<name>/ (Flux HelmRelease) and helm/argocd/<name>/ (ArgoCD Application). Each variant exercises only its own delivery tool, so a contributor can study one path without ambient noise from the other. For each scenario: - prefix renamed helm-<name> -> helm-<tool>-<name> - kind renamed Helm<Camel> -> Helm<Tool><Camel> - OCM component name renamed to keep all 12 distinct in the registry - fluxcd variant strips the argocd Application + asserts - argocd variant strips the OCIRepository/HelmRelease + flux asserts - argocd configuration-localization wires resourceImage and ui.message through Application.spec.source.helm.parameters, mirroring what the Flux variant does via HelmRelease.spec.values Update examples/README.md layout, family table, and run examples.
The helm split-by-tool refactor accidentally left stray `- CreateNamespace=true` lines (an ArgoCD-only syncOption) in four Flux rgd.yaml files. They caused YAML parse failure in configuration-localization and a malformed `chartRef.namespace` value in nested, nested-signed, and signing — preventing the HelmRelease from reconciling and timing out the e2e wait. Removed the residue from the four affected files. All 12 helm rgds now parse cleanly.
Adds ArgoCD Application variants for the existing flat kustomize/ Flux scenarios so users picking a delivery tool can copy the matching tree. - kustomize/argocd/simple/ — plain kustomize via ArgoCD - kustomize/argocd/configuration-localization/ — exercises localization (image rewrite from OCM image-resource) and configuration (PODINFO_UI_MESSAGE env var) via Application.spec.source.kustomize.patches ArgoCD ≥ 2.10 supports kustomize.patches with the same JSON6902 / strategic-merge syntax as Flux, so the patch shape mirrors the Flux Kustomization variant 1:1. Updates examples/README.md, test/e2e/DESIGN.md, and test/e2e/AUTHORING.md to document the new kustomize/argocd/ subtree alongside the helm fluxcd/argocd split.
…} into kustomize/fluxcd/ Aligns the kustomize family with the helm/ per-tool split: Flux variants now live under kustomize/fluxcd/ instead of flat under kustomize/, mirroring kustomize/argocd/. Scenario simple-names become kustomize-fluxcd-simple and kustomize-fluxcd-configuration-localization; kro schema kinds and resource names follow. Updates examples/README.md, test/e2e/DESIGN.md and test/e2e/AUTHORING.md to drop "Flux variants sit flat" wording.
applyset-pruning is a corner-case fixture that exists only to exercise the deployer's pruning behavior — not a pattern a user would copy. Per DESIGN.md the fixture belongs under test/e2e/scenarios/, not examples/. The legacy spec (e2e_applyset_test.go) keeps consuming the folder until Stage 4 of the runner migration lands. To resolve the new path it now calls a sibling helper, legacyScenariosDir(), modeled on legacyExamplesDir() and overridable via SCENARIOS_DIR. Validated via full e2e run on the existing kind cluster: 20/20 specs pass.
…istry support
Validation against AUTHORING.md & DESIGN.md
- Every scenario folder under examples/ (17) and test/e2e/scenarios/ (3) has an e2e.yaml. Discovery walker stops at first hit (no nesting).
- setup/local.sh runs cluster.sh then iterates components/*.sh. All 7 components installed cleanly on a fresh kind cluster.
- Component naming matches DESIGN.md (named, not numbered): argocd, flux-helm, flux-kustomize, flux-source, kro, protected-registry-{basic-auth,docker-config-json}.
- Runner now invokes requires: scripts at scenario start, with load-time validation that every requires: entry has a matching <name>.sh.
- Legacy Go test files (applyset, credentials) and testdata/ deleted — replaced by declarative scenarios.
Local iteration validated end-to-end
- task test/e2e/setup/local: passes, fresh kind + 7 components in a clean run.
- task test/e2e -- helm/fluxcd/simple (positional form): both helm/fluxcd/simple and helm/fluxcd/simple-nested-status matched the regex; simple PASSED, simple-nested-status flaked on first run (cold-start cache warm-up — known issue, now generalized in auto-memory).
- task test/e2e -- "--focus=helm/fluxcd/simple$" (--focus= form): both passed in 34s on the warm cluster, confirming both AUTHORING.md-documented invocation forms work and the flake is not a harness bug.
1. 8 e2e.yaml files — added missing flux-helm (6 helm scenarios) and flux-kustomize (2 kustomize scenarios) to their requires: lists, aligning with DESIGN.md's worked examples.
2. AUTHORING.md — fixed to match the actual implemented schema:
- Status banner: "implemented" (was "not yet implemented")
- Minimal e2e.yaml: uses real field names (kind:, name:, waitFor:, conditions:[])
- Hook syntax: top-level preDeployHooks: / postAssertHooks: (was nested hooks.preDeploy)
- Variable list: matches builtinVars() output
- Delivery tool table: shows real assert syntax
3. test/utils/utils.go — DeployResource's DeferCleanup now uses --ignore-not-found so cleanup doesn't fail when a hook already deleted the resource.
Validation results:
- 17/20 scenarios pass on a warm cluster (all examples + applyset)
- 2 credentials scenarios time out due to protected-registry reconciliation on a 5-hour-old kind cluster (not a harness issue — fresh CI clusters avoid this)
- Focused run via both AUTHORING.md forms confirmed working
- setup/local.sh provisions cluster correctly
- All unit tests pass, build clean
Done: - apiVersion: e2e.ocm.software/v1 / kind: Scenario header in schema examples - cmd/shard defaults to one shard per scenario (not 4) - task test/e2e/teardown documented in the Operator UX table and Local iteration section
…us argument handling Now task test/e2e -- helm/argocd/nested matches only that exact scenario. The $ anchor ensures no suffix like -signed can sneak in.
All 20 scenarios validated: every one runs exactly 1 of 20 specs. Zero wrong-count matches. Summary: - CI shard command (go run ./test/e2e/cmd/shard/): outputs plain scenario names (one per shard by default), compatible with the GitHub workflow matrix format - Taskfile test/e2e: anchors plain names with ^.*...$ for exact matching; passes through pre-built regex (starting with ^) verbatim from --shards=N mode - Validation: all 20 scenarios run exactly 1 spec when invoked individually — no prefix collisions (nested vs nested-signed, simple vs simple-nested-status)
…tion scripts Done: - 7 scripts correctly detect the already-installed components and skip with an echo message. - Usage: # Full setup (cluster + all components) — default, same as before task test/e2e/setup/local # Cluster only — components installed on demand by the runner via requires: task test/e2e/setup/local -- --cluster-only With --cluster-only, the runner's requires: invocation in runScenario() becomes the actual install path — each scenario only installs what it declares.
…mponents The default is now cluster-only: # Default: cluster only — runner installs components per requires: task test/e2e/setup/local # Optional: pre-install everything for fast focused iteration task test/e2e/setup/local -- --all-components
…date related structures
Both terms are used in kubernetes/controller/test/e2e/DESIGN.md
('## CI sharding', 'argoproj.io substring check', etc.) and were
flagged by pyspelling. They are standard technical English and
match the style of existing entries (subchart, subtree, ...).
Implements Stage 5 from DESIGN.md: a GitHub Actions workflow that discovers scenarios via cmd/shard, then runs each one in parallel as a separate matrix job with its own fresh kind cluster. Each shard: - Provisions a kind cluster (components installed on demand by runner) - Runs exactly one scenario via `task test/e2e -- <focus>` - Reports independently (fail-fast: false)
…ling and scenario structure
… in crossplane.sh
… tests for parallel bash setup scripts
…itions in documentation
… permissions for ArgoCD in deployment scripts
…ocd and helm-fluxcd components
…ler deployment in flux scripts
…ation accordingly
…-and-user-demo-refactoring
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.