diff --git a/.github/workflows/acceptance_test.yml b/.github/workflows/acceptance_test.yml new file mode 100644 index 00000000..f85b2ee6 --- /dev/null +++ b/.github/workflows/acceptance_test.yml @@ -0,0 +1,31 @@ +name: acceptance-test + +on: + push: + branches: + - main + pull_request: + types: + - opened + - synchronize + +permissions: + contents: read + +jobs: + acceptance-test: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout code + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + persist-credentials: false + + - name: Set up Go + uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0 + with: + go-version-file: "./go.mod" + + - name: Run acceptance tests + run: make test-acceptance COMPOSE="docker compose" diff --git a/.jscpd.json b/.jscpd.json new file mode 100644 index 00000000..e66435f0 --- /dev/null +++ b/.jscpd.json @@ -0,0 +1,6 @@ +{ + "ignore": [ + "vendor/**", + "**/vendor/**" + ] +} diff --git a/.mega-linter.yml b/.mega-linter.yml index 6b371be6..949fadcd 100644 --- a/.mega-linter.yml +++ b/.mega-linter.yml @@ -17,3 +17,6 @@ ENABLE_LINTERS: - REPOSITORY_KICS - YAML_YAMLLINT REPOSITORY_KICS_ARGUMENTS: "--fail-on high" +# Allow zizmor to use GITHUB_TOKEN for GitHub API access +ACTION_ZIZMOR_UNSECURED_ENV_VARIABLES: + - GITHUB_TOKEN diff --git a/AGENTS.md b/AGENTS.md index 05289804..9fd96c6d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,6 +50,10 @@ make test-behavioral # Devcontainer smoke test (verifies Containerfile builds) make test-devcontainer # → podman build -t complyctl-devcontainer-test .devcontainer/ + +# Acceptance tests (container-based, requires podman-compose or docker compose) +make test-acceptance +# → compose up --profile lifecycle (zot + seed + SUT containers) ``` ### Lint & Format @@ -86,6 +90,7 @@ make crapload-check # check for CRAP regressions against baseline | SonarCloud | `ci_sonarcloud.yml` | Code quality analysis | | Behavioral | `behavioral_assessment.yml` | Behavioral assessment reports | | Scheduled | `ci_scheduled.yml` | Daily OSV-Scanner and Scorecards | +| Acceptance Test | `acceptance_test.yml` | Container-based acceptance tests with real OCI registry | | Release | `release.yml` | Release automation | ## Project Structure @@ -125,6 +130,7 @@ plans/ # feature planning artifacts scripts/ # maintenance scripts (SPDX checks, workflow setup) specs/ # Speckit strategic specifications (NNN-*/ format) tests/ +├── acceptance/ # Container-based acceptance tests (compose + zot) ├── behavioral/ # behavioral test scenarios ├── cross-repo/ # cross-repo integration tests (complyctl + ampel + opa providers) ├── e2e/ # E2E tests (build-tag gated: -tags=e2e) @@ -151,6 +157,18 @@ vendor/ # vendored dependencies - **Line length**: 99 characters max unless exceeding improves readability +### Container Image Conventions + +- **Base images**: Red Hat UBI (`ubi10/ubi-minimal`) MUST + NOT be pinned by digest hash. Use `:latest` to track + current errata and security patches. Dismiss Scorecard + `Pinned-Dependencies` alerts for UBI images — this is + an intentional tradeoff favoring automatic errata + coverage over reproducibility. +- **Trivy suppressions**: Use `# trivy:ignore:DSXXX` + comments inline above the `FROM` line for accepted + findings on test/ephemeral containers. + ### Spec Writing Conventions - Use RFC 2119 language (MUST/SHOULD/MAY) for requirements @@ -286,6 +304,7 @@ packages organized by domain responsibility. ## Recent Changes - complypack-cache-versioning: `COMPLYTIME_CACHE_VERSIONS` env var configures retention count (default 1) for complypack cache versions per evaluator-id; `NewComplypackCache()` gains `*State` parameter for state-driven lookup and timestamp-based eviction ordering; `evictOldVersions()` becomes retention-count-aware (orphaned dirs first, then oldest by `LastUpdated`); `LookupByEvaluatorID()` resolves from state.json with directory-scan fallback; `SyncComplypack()` checks local cache before remote fetch with re-verification when verifier is configured, returns `(true, nil)` for cache hits to trigger generation invalidation; `EvaluatorIDToVersion()` reverse lookup on `*State`; `CacheRetentionCount()` in `internal/cache/retention.go`; `CheckComplypacks()` extended with `walkCacheSize()` and `findOrphanedVersions()` for cache health reporting; `complyctl doctor` reports cache size and orphaned/untracked versions (#676) - per-entry-verification: Per-entry `verification:` and `skip_verify:` fields on `PolicyEntry` in `internal/complytime/config.go` for policy-level signature verification overrides; `resolveVerifier()` in `cmd/complyctl/cli/get.go` with verifier cache and resolution priority chain (entry → workspace → none); error collection in `syncAllPolicies()`/`syncAllComplypacks()` via `errors.Join()` for partial-failure resilience; cross-group error collection in `syncAll()` so policy failures do not block complypack sync; per-entry WARNING to stderr on sync failure for real-time feedback; `validateEntries()` extended for per-entry verification validation and mutual exclusivity of `verification:` and `skip_verify:` (#680) +- acceptance-tests: Container-based acceptance test stack (`tests/acceptance/`, `make test-acceptance`, `acceptance_test.yml`); zot + seed + sut compose architecture validates real OCI registry interop; `build-acceptance-test` target compiles acceptance binary; `test-acceptance-clean` tears down containers and volumes; `SaveState` atomic write refactor in `internal/cache/state.go` - debug-visible-output: `--debug` flag now tees all log messages to stderr via `teeWriter` in `pkg/log/log.go`; `enableDebug()` in `cmd/complyctl/cli/root.go` reconstructs logger with `NewTeeWriter(stderr, logFile)` and forces `termenv.ANSI256` color profile when stderr is a TTY (respects `NO_COLOR`); `Debug log: ` hint printed to stderr after workspace resolution; flag description updated to `"output debug logs to stderr and log file"`; `SetColorProfile()` method added to `CharmHclog` adapter (#614) - sigstore-verification: `complyctl get` verifies OCI artifact signatures via `sigstore-go` when `verification:` configured in `complytime.yaml`; keyless (OIDC issuer + identity) and keyed (public key) modes; pre-copy verification via registry API before `oras.Copy()`; `--skip-verify` flag; `VerificationConfig` in `WorkspaceConfig`; `VerifyFunc`/`NewKeylessVerifier`/`NewKeyedVerifier` in `internal/cache/verify.go`; `PolicyState` gains `Verified`/`SignerIdentity`/`Issuer`/`VerifiedAt` fields; `SyncOption`/`WithVerifier()` functional options on `Sync`/`ComplypackSync`; `complyctl list` VERIFIED column; `complyctl doctor` `CheckVerification` diagnostic; `sigstore-go` v1.2.1 + `go-containerregistry` dependencies added - markdown-report-redesign: `--format pretty` markdown report redesigned with summary metadata table, pass rate, grouped controls table (control + requirement rows with message column), findings section grouped by result type with recommendation and collapsible evidence per finding, evaluation log in collapsible `
`; `Evidence` message and `recommendation` field added to proto `AssessmentLog`; `provider.Evidence` type added to `pkg/provider/client.go`; evaluator populates `gemara.Evidence` and `Recommendation` on assessment logs; `internal/output/markdown.go` rewritten with `writeSummary`/`writeControlsTable`/`writeFindings`/`writeEvaluationLog` methods diff --git a/CHANGELOG.md b/CHANGELOG.md index c8532187..17540a41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -58,6 +58,10 @@ error. Verifiers are cached by configuration to avoid redundant construction for entries sharing the same verification settings. (#680) +- Container-based acceptance tests (`make test-acceptance`): runs + complyctl against a real zot OCI registry in a compose stack, + validating OCI artifact interop that in-process mock registries + cannot catch (#653) - Redesigned markdown report (`--format pretty`) with summary metadata table, pass rate, grouped controls table with messages, findings section grouped by result type with recommendation and collapsible diff --git a/Makefile b/Makefile index 9fd33f10..4f764373 100644 --- a/Makefile +++ b/Makefile @@ -13,6 +13,18 @@ GO_LD_EXTRAFLAGS := -X github.com/complytime/complyctl/internal/version.version= MAN_COMPLYCTL = docs/man/complyctl.md MAN_COMPLYCTL_OUTPUT = docs/man/complyctl.1 +COMPOSE ?= $(shell command -v podman-compose 2>/dev/null || echo "docker compose") + +# Derive CONTAINER_ENGINE from COMPOSE to avoid cross-engine mismatches +# (e.g. docker compose creates containers, but podman ps cannot see them). +ifeq ($(findstring docker,$(COMPOSE)),docker) +CONTAINER_ENGINE ?= docker +else +CONTAINER_ENGINE ?= $(shell command -v podman 2>/dev/null || echo docker) +endif + +.DEFAULT_GOAL := help + ##@ Proto proto: ## generate protobuf code (requires buf) @@ -62,6 +74,29 @@ test-devcontainer: ## verify devcontainer Containerfile builds @echo "Containerfile builds successfully." .PHONY: test-devcontainer +ACCEPTANCE_COMPOSE = $(COMPOSE) -f tests/acceptance/compose.yaml --profile lifecycle + +test-acceptance: build build-test-provider build-acceptance-test ## run container-based acceptance tests (requires podman-compose or docker compose) + @$(ACCEPTANCE_COMPOSE) up --build -d || { $(ACCEPTANCE_COMPOSE) down -v ; exit 1 ; } ; \ + sut=$$($(CONTAINER_ENGINE) ps -aq --filter name=sut | head -1) ; \ + if [ -z "$$sut" ]; then \ + echo "ERROR: SUT container not found" ; \ + $(ACCEPTANCE_COMPOSE) down -v ; \ + exit 1 ; \ + fi ; \ + rc=$$($(CONTAINER_ENGINE) wait $$sut) ; \ + echo "=== seed logs ===" ; \ + $(ACCEPTANCE_COMPOSE) logs seed 2>&1 || true ; \ + echo "=== sut logs ===" ; \ + $(CONTAINER_ENGINE) logs $$sut ; \ + $(ACCEPTANCE_COMPOSE) down -v ; \ + exit $$rc +.PHONY: test-acceptance + +test-acceptance-clean: ## tear down acceptance test containers and volumes + $(ACCEPTANCE_COMPOSE) down -v --remove-orphans +.PHONY: test-acceptance-clean + ##@ Compilation all: clean vendor test-unit build ## compile from scratch @@ -75,6 +110,10 @@ build-test-provider: prep-build-dir ## build test provider for E2E tests go build -mod=vendor -o $(GO_BUILD_BINDIR)/complyctl-provider-test ./cmd/test-provider .PHONY: build-test-provider +build-acceptance-test: prep-build-dir ## compile acceptance test binary + go test -c -tags=acceptance -mod=vendor -o $(GO_BUILD_BINDIR)/acceptance.test ./tests/acceptance/ +.PHONY: build-acceptance-test + build-behavioral-report: prep-build-dir ## build behavioral report tool (go test -json -> EvaluationLog + SARIF) go build -mod=vendor -o $(GO_BUILD_BINDIR)/behavioral-report ./cmd/behavioral-report .PHONY: build-behavioral-report diff --git a/README.md b/README.md index 821e1345..00683d9f 100644 --- a/README.md +++ b/README.md @@ -170,7 +170,6 @@ complyctl scan --policy-id nist-800-53-r5 complyctl scan prod --format oscal complyctl scan --policy-id nist-800-53-r5 --format pretty complyctl scan --policy-id nist-800-53-r5 --format sarif - ``` | Argument / Flag | Short | Description | diff --git a/docs/TESTING_ENVIRONMENT.md b/docs/TESTING_ENVIRONMENT.md index f1fca987..afc0dc2c 100644 --- a/docs/TESTING_ENVIRONMENT.md +++ b/docs/TESTING_ENVIRONMENT.md @@ -382,6 +382,49 @@ After resuming, restart the mock registry manually: ./bin/mock-oci-registry & ``` +## Container-Based Acceptance Tests + +The acceptance test suite validates real OCI registry interop using +a three-container compose stack. Unlike the devcontainer environment, +acceptance tests are automated and run in CI on every PR. + +### Prerequisites + +You need `podman-compose` or `docker compose` (v2) and a container +runtime (`podman` or `docker`). + +### Running + +```bash +# Build binaries and run the full acceptance suite +make test-acceptance + +# Specify a different compose command (default: podman-compose) +make test-acceptance COMPOSE="docker compose" + +# Tear down containers and volumes after a failed run +make test-acceptance-clean +``` + +### Architecture + +The compose stack (`tests/acceptance/compose.yaml`) runs three +services under the `lifecycle` profile: + +- **zot** -- a real OCI-compliant registry (Project Zot) listening + on port 5000 inside the compose network. +- **seed** -- a short-lived container that uses the `oras` CLI to + push Gemara test policies into zot, then exits. The seed service + must complete successfully before the SUT starts. +- **sut** (system under test) -- runs the acceptance test binary + (`go test -tags=acceptance ./tests/acceptance/...`) against the + live registry. The compose stack exits with the SUT's exit code. + +### CI + +The `acceptance_test.yml` workflow runs `make test-acceptance` on +every push and PR using `docker compose`. + ## See Also - [Quick Start](./QUICK_START.md) diff --git a/internal/cache/state.go b/internal/cache/state.go index 7f003afa..049d3411 100644 --- a/internal/cache/state.go +++ b/internal/cache/state.go @@ -72,7 +72,9 @@ func initStateMaps(s *State) { } } -// SaveState writes the state to state.json in the given cache directory. +// SaveState atomically writes the state to state.json in the given cache +// directory. It marshals to a sibling temp file then renames it into place so +// concurrent readers never observe a truncated or partial write. func SaveState(state *State, cacheDir string) error { if err := os.MkdirAll(cacheDir, 0755); err != nil { return fmt.Errorf("failed to create cache directory: %w", err) @@ -85,7 +87,32 @@ func SaveState(state *State, cacheDir string) error { return fmt.Errorf("failed to marshal state: %w", err) } - if err := os.WriteFile(statePath, data, 0600); err != nil { + // Write to a temp file in the same directory so os.Rename is atomic + // (same filesystem, POSIX guarantee). This prevents concurrent readers + // from observing a truncated file mid-write. + tmp, err := os.CreateTemp(cacheDir, ".state-*.json.tmp") + if err != nil { + return fmt.Errorf("failed to create temp state file: %w", err) + } + tmpPath := tmp.Name() + + if _, err := tmp.Write(data); err != nil { + _ = tmp.Close() + _ = os.Remove(tmpPath) + return fmt.Errorf("failed to write temp state file %s: %w", tmpPath, err) + } + if err := tmp.Chmod(0600); err != nil { + _ = tmp.Close() + _ = os.Remove(tmpPath) + return fmt.Errorf("failed to set permissions on temp state file: %w", err) + } + if err := tmp.Close(); err != nil { + _ = os.Remove(tmpPath) + return fmt.Errorf("failed to close temp state file: %w", err) + } + + if err := os.Rename(tmpPath, statePath); err != nil { + _ = os.Remove(tmpPath) return fmt.Errorf("failed to write state file %s: %w", statePath, err) } diff --git a/internal/policy/generation_state.go b/internal/policy/generation_state.go index 1eaa0feb..c3f97c3f 100644 --- a/internal/policy/generation_state.go +++ b/internal/policy/generation_state.go @@ -112,7 +112,7 @@ func InvalidateForEvaluator(baseDir, evaluatorID string) (warnings []string, _ e if os.IsNotExist(rootErr) { return nil, nil } - return nil, rootErr + return nil, fmt.Errorf("failed to open generation directory: %w", rootErr) } defer root.Close() diff --git a/tests/acceptance/Dockerfile.seed b/tests/acceptance/Dockerfile.seed new file mode 100644 index 00000000..4f939f6e --- /dev/null +++ b/tests/acceptance/Dockerfile.seed @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: Apache-2.0 +# trivy:ignore:DS001 -- test container, intentionally floats on latest UBI +# trivy:ignore:DS026 -- ephemeral seed container, exits after seeding registry +# scorecard:pinned-dependencies -- intentionally unpinned; UBI images track latest for errata coverage +FROM registry.access.redhat.com/ubi10/ubi-minimal:latest + +ARG ORAS_VERSION=1.2.2 +# sha256 from https://github.com/oras-project/oras/releases/tag/v1.2.2 checksums.txt +ARG ORAS_SHA256=bff970346470e5ef888e9f2c0bf7f8ee47283f5a45207d6e7a037da1fb0eae0d + +RUN microdnf install -y tar gzip && microdnf clean all + +ADD https://github.com/oras-project/oras/releases/download/v${ORAS_VERSION}/oras_${ORAS_VERSION}_linux_amd64.tar.gz /tmp/oras.tar.gz +RUN echo "${ORAS_SHA256} /tmp/oras.tar.gz" | sha256sum -c - && \ + tar -xzf /tmp/oras.tar.gz -C /usr/local/bin oras && \ + rm /tmp/oras.tar.gz && \ + chmod +x /usr/local/bin/oras + +COPY seed.sh /seed.sh +RUN chmod +x /seed.sh + +USER nobody +ENTRYPOINT ["/seed.sh"] diff --git a/tests/acceptance/Dockerfile.sut b/tests/acceptance/Dockerfile.sut new file mode 100644 index 00000000..53e016e3 --- /dev/null +++ b/tests/acceptance/Dockerfile.sut @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: Apache-2.0 + +# Runtime-only container for acceptance tests. +# All binaries are built locally and injected from bin/. +# trivy:ignore:DS001 -- test container, intentionally floats on latest UBI +# trivy:ignore:DS026 -- ephemeral test container, compose healthcheck covers readiness +# scorecard:pinned-dependencies -- intentionally unpinned; UBI images track latest for errata coverage +FROM registry.access.redhat.com/ubi10/ubi-minimal:latest + +RUN microdnf install -y shadow-utils && microdnf clean all && \ + useradd -m testuser + +COPY bin/acceptance.test /usr/local/bin/acceptance.test +COPY bin/complyctl /usr/local/bin/complyctl +COPY bin/complyctl-provider-test \ + /home/testuser/.complytime/providers/complyctl-provider-test + +RUN chown -R testuser:testuser /home/testuser/.complytime && \ + chmod 755 /home/testuser/.complytime/providers/complyctl-provider-test + +USER testuser +WORKDIR /home/testuser + +ENTRYPOINT ["/usr/local/bin/acceptance.test"] +CMD ["-test.v", "-test.timeout=3m"] diff --git a/tests/acceptance/Dockerfile.sut.dockerignore b/tests/acceptance/Dockerfile.sut.dockerignore new file mode 100644 index 00000000..4b410f63 --- /dev/null +++ b/tests/acceptance/Dockerfile.sut.dockerignore @@ -0,0 +1,6 @@ +# SUT container only needs pre-built binaries from bin/. +# Exclude everything else. +* +!bin/acceptance.test +!bin/complyctl +!bin/complyctl-provider-test diff --git a/tests/acceptance/acceptance_test.go b/tests/acceptance/acceptance_test.go new file mode 100644 index 00000000..1221950e --- /dev/null +++ b/tests/acceptance/acceptance_test.go @@ -0,0 +1,203 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build acceptance + +// Container-based acceptance tests for complyctl exercising real OCI registry +// interop. These tests run inside a container orchestrated by compose.yaml +// against a real zot registry seeded with Gemara policies via oras CLI. +// +// Run: +// +// make test-acceptance +// +// Or manually: +// +// podman-compose -f tests/acceptance/compose.yaml up --profile lifecycle --build \ +// --abort-on-container-exit --exit-code-from sut +package acceptance + +import ( + "fmt" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/goccy/go-yaml" +) + +// TestMain runs a preflight check to verify the registry is reachable and +// seeded before executing any tests. This gives a clear error message when +// the seed container failed silently. +func TestMain(m *testing.M) { + registryURL := os.Getenv("REGISTRY_URL") + policyID := os.Getenv("TEST_POLICY_ID") + targetID := os.Getenv("TEST_TARGET_ID") + if registryURL == "" || policyID == "" || targetID == "" { + fmt.Fprintln(os.Stderr, "FATAL: REGISTRY_URL, TEST_POLICY_ID, and TEST_TARGET_ID must be set") + os.Exit(1) + } + + // Verify registry is reachable + client := &http.Client{Timeout: 10 * time.Second} + resp, err := client.Get(registryURL + "/v2/") + if err != nil { + fmt.Fprintf(os.Stderr, "FATAL: registry not reachable at %s: %v\n", registryURL, err) + os.Exit(1) + } + resp.Body.Close() + + // Verify policy was seeded (check tags endpoint) + resp, err = client.Get(registryURL + "/v2/policies/" + policyID + "/tags/list") + if err != nil || resp.StatusCode != http.StatusOK { + fmt.Fprintf(os.Stderr, "FATAL: policy policies/%s not found in registry — seed container likely failed\n", policyID) + if err == nil { + resp.Body.Close() + } + os.Exit(1) + } + resp.Body.Close() + + os.Exit(m.Run()) +} + +// TestAcceptance_OCILifecycle exercises the full happy-path lifecycle against a +// real zot OCI registry: write config -> get -> doctor -> generate -> scan -> +// verify evaluation log. +func TestAcceptance_OCILifecycle(t *testing.T) { + env := loadTestEnv(t) + workDir := t.TempDir() + writeWorkspaceConfig(t, workDir, env) + + // get: fetch policy from zot into OCI layout cache + out := runComplyctl(t, workDir, "get", "--workspace", workDir) + assert.Contains(t, out, "Synchronization completed.") + + homeDir, err := os.UserHomeDir() + require.NoError(t, err) + cacheDir := filepath.Join(homeDir, ".complytime", "policies") + assert.DirExists(t, cacheDir, "policy cache directory must exist after get") + + // doctor: verify all checks pass + out = runComplyctl(t, workDir, "doctor", "--workspace", workDir) + assert.Contains(t, out, "passed") + assert.Contains(t, out, "0 failed") + + // generate: resolve policy graph and invoke test provider + runComplyctl(t, workDir, "generate", + "--policy-id", env.PolicyID, "--workspace", workDir) + + // scan: produce evaluation log + out = runComplyctl(t, workDir, "scan", env.TargetID, "--workspace", workDir) + assert.Contains(t, out, "requirements:") + + // verify evaluation log + scanDir := filepath.Join(workDir, ".complytime", "scan") + evalFile := assertOutputFile(t, scanDir, "evaluation-log-", ".yaml") + + data, err := os.ReadFile(evalFile) + require.NoError(t, err) + + var evalLog map[string]interface{} + require.NoError(t, yaml.Unmarshal(data, &evalLog)) + + // Check the evaluation log contains the expected requirement ID + evaluations, ok := evalLog["evaluations"].([]interface{}) + require.True(t, ok, "evaluation log must have evaluations array") + require.NotEmpty(t, evaluations, "evaluations must not be empty") + + foundRequirement := false + for _, eval := range evaluations { + evalMap, ok := eval.(map[string]interface{}) + if !ok { + continue + } + logs, ok := evalMap["assessment-logs"].([]interface{}) + if !ok { + continue + } + for _, log := range logs { + logMap, ok := log.(map[string]interface{}) + if !ok { + continue + } + req, ok := logMap["requirement"].(map[string]interface{}) + if !ok { + continue + } + if req["entry-id"] == "block-force-push" { + foundRequirement = true + result, _ := logMap["result"].(string) + assert.Equal(t, "Passed", result, + "block-force-push requirement must pass") + } + } + } + assert.True(t, foundRequirement, + "evaluation log must contain block-force-push requirement") + + // Verify target ID in evaluation log + target, ok := evalLog["target"].(map[string]interface{}) + require.True(t, ok, "evaluation log must have target") + assert.Equal(t, env.TargetID, target["id"], + "evaluation log target must match configured target ID") +} + +// TestAcceptance_FormatInterop validates that artifacts pushed by the oras CLI +// are correctly consumed by complyctl's oras-go client. Verifies the OSCAL +// assessment-results output contains correct control and requirement IDs +// from the seeded policy. +func TestAcceptance_FormatInterop(t *testing.T) { + env := loadTestEnv(t) + workDir := t.TempDir() + writeWorkspaceConfig(t, workDir, env) + + // get + generate (prerequisites for scan) + runComplyctl(t, workDir, "get", "--workspace", workDir) + runComplyctl(t, workDir, "generate", + "--policy-id", env.PolicyID, "--workspace", workDir) + + // scan with OSCAL output + out := runComplyctl(t, workDir, "scan", env.TargetID, + "--format", "oscal", "--workspace", workDir) + assert.Contains(t, out, "requirements:") + + // verify OSCAL output + scanDir := filepath.Join(workDir, ".complytime", "scan") + assertOutputFile(t, scanDir, "evaluation-log-", ".yaml") + oscalFile := assertOutputFile(t, scanDir, "assessment-results-", ".json") + + findings := parseOSCALFindings(t, oscalFile) + require.NotEmpty(t, findings, "OSCAL output must contain findings") + + // Verify the block-force-push requirement appears in findings + foundBlockForcePush := false + for _, finding := range findings { + target, ok := finding["target"].(map[string]interface{}) + if !ok { + continue + } + targetID, _ := target["target-id"].(string) + if targetID == "block-force-push" { + foundBlockForcePush = true + + // Verify result state + status, ok := target["status"].(map[string]interface{}) + require.True(t, ok, "finding target must have status") + assert.Equal(t, "satisfied", status["state"], + "block-force-push must be satisfied") + + // Verify finding title format + title, _ := finding["title"].(string) + assert.True(t, strings.Contains(title, "block-force-push"), + "finding title must reference block-force-push") + } + } + assert.True(t, foundBlockForcePush, + "OSCAL findings must contain block-force-push requirement") +} diff --git a/tests/acceptance/compose.yaml b/tests/acceptance/compose.yaml new file mode 100644 index 00000000..79a4ec7a --- /dev/null +++ b/tests/acceptance/compose.yaml @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: Apache-2.0 +# +# Container-based acceptance tests for complyctl. +# Usage: make test-acceptance +# +# Architecture: +# zot — real OCI registry (CNCF zot-minimal) +# seed — init container that pushes Gemara testdata into zot via oras CLI +# sut — compiled Go test binary exercising complyctl against zot + +services: + zot: + image: ghcr.io/project-zot/zot-minimal-linux-amd64:v2.1.18@sha256:f1ffb7a5bbddc0feea83646e29c587ecf39b3193733b447749d4c9ead111a395 + profiles: [lifecycle] + volumes: + - ./zot-config.json:/etc/zot/config.json:ro + command: ["serve", "/etc/zot/config.json"] + + seed: + build: + context: . + dockerfile: Dockerfile.seed + profiles: [lifecycle] + depends_on: + zot: + condition: service_started + volumes: + - ./testdata:/testdata:ro + environment: + REGISTRY_URL: "zot:5000" + + sut: + build: + context: ../../ + dockerfile: tests/acceptance/Dockerfile.sut + profiles: [lifecycle] + depends_on: + seed: + condition: service_completed_successfully + environment: + REGISTRY_URL: "http://zot:5000" + TEST_POLICY_ID: "acceptance-test" + TEST_TARGET_ID: "acceptance-target" diff --git a/tests/acceptance/helpers_test.go b/tests/acceptance/helpers_test.go new file mode 100644 index 00000000..00697fe4 --- /dev/null +++ b/tests/acceptance/helpers_test.go @@ -0,0 +1,140 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build acceptance + +package acceptance + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const ( + complyctlBinary = "/usr/local/bin/complyctl" +) + +// testEnv holds configuration sourced from environment variables +// injected by compose.yaml. +type testEnv struct { + RegistryURL string + PolicyID string + TargetID string +} + +// loadTestEnv reads required environment variables. Fails the test if +// any required variable is missing. +func loadTestEnv(t *testing.T) testEnv { + t.Helper() + registryURL := os.Getenv("REGISTRY_URL") + require.NotEmpty(t, registryURL, "REGISTRY_URL must be set") + policyID := os.Getenv("TEST_POLICY_ID") + require.NotEmpty(t, policyID, "TEST_POLICY_ID must be set") + targetID := os.Getenv("TEST_TARGET_ID") + require.NotEmpty(t, targetID, "TEST_TARGET_ID must be set") + return testEnv{ + RegistryURL: registryURL, + PolicyID: policyID, + TargetID: targetID, + } +} + +// writeWorkspaceConfig creates .complytime/complytime.yaml in dir +// with the given registry URL, policy ID, and target ID. +func writeWorkspaceConfig(t *testing.T, dir string, env testEnv) { + t.Helper() + configYAML := fmt.Sprintf(`policies: + - url: %s/policies/%s + id: %s +variables: + workspace: %s +targets: + - id: %s + policies: + - %s + variables: + env: acceptance +`, env.RegistryURL, env.PolicyID, env.PolicyID, dir, env.TargetID, env.PolicyID) + + configDir := filepath.Join(dir, ".complytime") + require.NoError(t, os.MkdirAll(configDir, 0o755)) + require.NoError(t, os.WriteFile( + filepath.Join(configDir, "complytime.yaml"), + []byte(configYAML), 0o644)) +} + +// runComplyctl executes the complyctl binary with given args and returns +// combined stdout+stderr. Fails the test on non-zero exit code. +func runComplyctl(t *testing.T, workDir string, args ...string) string { + t.Helper() + cmd := exec.Command(complyctlBinary, args...) + cmd.Dir = workDir + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("complyctl %s failed:\n%s\nerror: %v", + strings.Join(args, " "), string(out), err) + } + return string(out) +} + +// assertOutputFile finds a file matching prefix+suffix in dir. Returns +// its path. Fails the test if no matching file exists or is empty. +func assertOutputFile(t *testing.T, dir, prefix, suffix string) string { + t.Helper() + entries, err := os.ReadDir(dir) + require.NoError(t, err, "output directory %s must be readable", dir) + for _, e := range entries { + name := e.Name() + if strings.HasPrefix(name, prefix) && strings.HasSuffix(name, suffix) { + path := filepath.Join(dir, name) + assert.FileExists(t, path) + info, statErr := os.Stat(path) + require.NoError(t, statErr) + assert.Greater(t, info.Size(), int64(0), "%s must not be empty", name) + return path + } + } + t.Fatalf("no file matching %s*%s found in %s (contents: %v)", + prefix, suffix, dir, entries) + return "" +} + +// parseOSCALFindings reads an OSCAL assessment-results JSON file and +// returns the findings array from the first result. Fails the test if +// the structure is invalid. +func parseOSCALFindings(t *testing.T, path string) []map[string]interface{} { + t.Helper() + data, err := os.ReadFile(path) + require.NoError(t, err) + + var doc map[string]interface{} + require.NoError(t, json.Unmarshal(data, &doc)) + + ar, ok := doc["assessment-results"].(map[string]interface{}) + require.True(t, ok, "must have assessment-results root key") + + results, ok := ar["results"].([]interface{}) + require.True(t, ok, "must have results array") + require.NotEmpty(t, results, "results must not be empty") + + result0, ok := results[0].(map[string]interface{}) + require.True(t, ok, "first result must be an object") + + findingsRaw, ok := result0["findings"].([]interface{}) + require.True(t, ok, "first result must have findings") + + findings := make([]map[string]interface{}, 0, len(findingsRaw)) + for _, f := range findingsRaw { + fm, ok := f.(map[string]interface{}) + require.True(t, ok, "each finding must be an object") + findings = append(findings, fm) + } + return findings +} diff --git a/tests/acceptance/seed.sh b/tests/acceptance/seed.sh new file mode 100644 index 00000000..aa0c0524 --- /dev/null +++ b/tests/acceptance/seed.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: Apache-2.0 +set -euo pipefail + +REGISTRY="${REGISTRY_URL:-zot:5000}" + +echo "Waiting for registry at ${REGISTRY}..." +for i in $(seq 1 30); do + if oras repo list --plain-http "${REGISTRY}" >/dev/null 2>&1; then + echo "Registry is ready." + break + fi + if [[ "$i" -eq 30 ]]; then + echo "ERROR: Registry not ready after 30 attempts." + exit 1 + fi + sleep 1 +done + +echo "Pushing Gemara policy bundle..." +cd /testdata +oras push --plain-http "${REGISTRY}/policies/acceptance-test:v1.0.0" \ + catalog.yaml:application/vnd.gemara.catalog.v1+yaml \ + policy.yaml:application/vnd.gemara.policy.v1+yaml + +echo "Tagging latest..." +oras tag --plain-http "${REGISTRY}/policies/acceptance-test:v1.0.0" latest + +echo "Verifying push..." +tags=$(oras repo tags --plain-http "${REGISTRY}/policies/acceptance-test") +echo "Tags: ${tags}" + +if ! echo "${tags}" | grep -q "v1.0.0"; then + echo "ERROR: v1.0.0 tag not found after push." + exit 1 +fi +if ! echo "${tags}" | grep -q "latest"; then + echo "ERROR: latest tag not found after tagging." + exit 1 +fi + +echo "Seed complete." diff --git a/tests/acceptance/testdata/catalog.yaml b/tests/acceptance/testdata/catalog.yaml new file mode 100644 index 00000000..91ae8b7e --- /dev/null +++ b/tests/acceptance/testdata/catalog.yaml @@ -0,0 +1,30 @@ +title: Acceptance Test Controls +controls: + - id: force-push-protection + title: Restrict Force Pushes + objective: Force pushes to protected branches must be blocked + group: source-code + assessment-requirements: + - id: block-force-push + text: Force pushes to protected branches must be blocked + applicability: + - github-repos + state: Active + state: Active +groups: + - id: source-code + title: Source Code Protection + description: Controls for source code repository protection +metadata: + id: acceptance-test-controls + type: ControlCatalog + gemara-version: 1.3.0 + description: Minimal catalog for container-based acceptance testing + author: + id: complytime + name: ComplyTime + type: Software Assisted + applicability-groups: + - id: github-repos + title: GitHub Repositories + description: Applicable to GitHub-hosted repositories diff --git a/tests/acceptance/testdata/policy.yaml b/tests/acceptance/testdata/policy.yaml new file mode 100644 index 00000000..aef689d4 --- /dev/null +++ b/tests/acceptance/testdata/policy.yaml @@ -0,0 +1,49 @@ +title: Acceptance Test Policy +metadata: + id: acceptance-test-policy + type: Policy + gemara-version: 1.3.0 + description: Policy for container-based acceptance testing using the test provider + author: + id: complytime + name: ComplyTime + type: Software Assisted + mapping-references: + - id: acceptance-test-controls + title: Acceptance Test Controls + version: 1.0.0 + description: Minimal catalog for acceptance testing +contacts: + responsible: + - name: Test Runner + accountable: + - name: Test Runner +scope: + in: + technologies: + - testing +imports: + catalogs: + - reference-id: acceptance-test-controls +adherence: + evaluation-methods: + - id: behavioral-eval + type: Behavioral + mode: Automated + description: Automated acceptance test evaluation + executor: + id: test + name: test-evaluator + type: Software + assessment-plans: + - id: block-force-push + requirement-id: block-force-push + frequency: on-demand + evaluation-methods: + - id: behavioral-eval-plan + type: Behavioral + mode: Automated + executor: + id: test + name: test-evaluator + type: Software diff --git a/tests/acceptance/zot-config.json b/tests/acceptance/zot-config.json new file mode 100644 index 00000000..75130f23 --- /dev/null +++ b/tests/acceptance/zot-config.json @@ -0,0 +1,12 @@ +{ + "storage": { + "rootDirectory": "/var/lib/registry" + }, + "http": { + "address": "0.0.0.0", + "port": "5000" + }, + "log": { + "level": "error" + } +}