From 2d819ff55d785aa97d31b7500a1eb760555a2214 Mon Sep 17 00:00:00 2001 From: Em Date: Fri, 10 Jul 2026 14:40:04 +0100 Subject: [PATCH] feat: add emoji status indicators to markdown report for CI step summary - Add resultEmoji() helper mapping gemara.Result to complytime.Status* emoji constants (StatusPassed, StatusFailed, StatusSkipped, StatusError) - Prefix Controls table Result column with emoji for control and requirement rows in writeControlsTable() - Prefix summary counts table headers with emoji in writeSummary() - Prefix findings group headers with emoji in writeFindings() - Add table-driven TestResultEmoji unit test covering all 6 result types plus default case - Add 3 new integration tests with section-scoped assertions - Update 6 existing tests for emoji presence - Update upstream TestMarkdown_NeedsReviewInFindings for emoji prefix - Add OpenSpec change artifacts (proposal, design, spec, tasks) Closes #697 --- AGENTS.md | 1 + CHANGELOG.md | 5 + internal/output/markdown.go | 29 +- internal/output/markdown_internal_test.go | 34 +++ internal/output/markdown_test.go | 247 +++++++++++++++++- .../ci-step-summary-report/.openspec.yaml | 2 + .../changes/ci-step-summary-report/design.md | 111 ++++++++ .../ci-step-summary-report/proposal.md | 83 ++++++ .../specs/ci-scannable-report/spec.md | 130 +++++++++ .../changes/ci-step-summary-report/tasks.md | 45 ++++ 10 files changed, 676 insertions(+), 11 deletions(-) create mode 100644 internal/output/markdown_internal_test.go create mode 100644 openspec/changes/ci-step-summary-report/.openspec.yaml create mode 100644 openspec/changes/ci-step-summary-report/design.md create mode 100644 openspec/changes/ci-step-summary-report/proposal.md create mode 100644 openspec/changes/ci-step-summary-report/specs/ci-scannable-report/spec.md create mode 100644 openspec/changes/ci-step-summary-report/tasks.md diff --git a/AGENTS.md b/AGENTS.md index 05289804a..46d6dd862 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -284,6 +284,7 @@ packages organized by domain responsibility. ## Recent Changes +- ci-step-summary-report: Emoji status indicators added to `--format pretty` markdown report; `resultEmoji()` helper maps `gemara.Result` to `complytime.Status*` constants; `writeControlsTable()` prefixes Result column with emoji for control and requirement rows; `writeSummary()` prefixes counts table headers with emoji; `writeFindings()` prefixes group headers with emoji; report now directly usable as GitHub Actions Step Summary via `cat report-*.md >> $GITHUB_STEP_SUMMARY` (#697) - complypack-cache-versioning: `COMPLYTIME_CACHE_VERSIONS` env var configures retention count (default 1) for complypack cache versions per evaluator-id; `NewComplypackCache()` gains `*State` parameter for state-driven lookup and timestamp-based eviction ordering; `evictOldVersions()` becomes retention-count-aware (orphaned dirs first, then oldest by `LastUpdated`); `LookupByEvaluatorID()` resolves from state.json with directory-scan fallback; `SyncComplypack()` checks local cache before remote fetch with re-verification when verifier is configured, returns `(true, nil)` for cache hits to trigger generation invalidation; `EvaluatorIDToVersion()` reverse lookup on `*State`; `CacheRetentionCount()` in `internal/cache/retention.go`; `CheckComplypacks()` extended with `walkCacheSize()` and `findOrphanedVersions()` for cache health reporting; `complyctl doctor` reports cache size and orphaned/untracked versions (#676) - per-entry-verification: Per-entry `verification:` and `skip_verify:` fields on `PolicyEntry` in `internal/complytime/config.go` for policy-level signature verification overrides; `resolveVerifier()` in `cmd/complyctl/cli/get.go` with verifier cache and resolution priority chain (entry → workspace → none); error collection in `syncAllPolicies()`/`syncAllComplypacks()` via `errors.Join()` for partial-failure resilience; cross-group error collection in `syncAll()` so policy failures do not block complypack sync; per-entry WARNING to stderr on sync failure for real-time feedback; `validateEntries()` extended for per-entry verification validation and mutual exclusivity of `verification:` and `skip_verify:` (#680) - debug-visible-output: `--debug` flag now tees all log messages to stderr via `teeWriter` in `pkg/log/log.go`; `enableDebug()` in `cmd/complyctl/cli/root.go` reconstructs logger with `NewTeeWriter(stderr, logFile)` and forces `termenv.ANSI256` color profile when stderr is a TTY (respects `NO_COLOR`); `Debug log: ` hint printed to stderr after workspace resolution; flag description updated to `"output debug logs to stderr and log file"`; `SetColorProfile()` method added to `CharmHclog` adapter (#614) diff --git a/CHANGELOG.md b/CHANGELOG.md index c85321878..a856c9dc5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,11 @@ ### Added +- Emoji status indicators in markdown report (`--format pretty`): Controls + table rows, summary counts headers, and findings group headers now display + emoji prefixes (`✅`, `❌`, `⏭️`, `⚠️`) matching the terminal scan + summary. Makes the report directly usable as a GitHub Actions Step Summary + via `cat report-*.md >> $GITHUB_STEP_SUMMARY` (#697) - Complypack cache version management: `COMPLYTIME_CACHE_VERSIONS` environment variable configures how many complypack versions to retain per evaluator-id in the local cache (default: 1, preserving diff --git a/internal/output/markdown.go b/internal/output/markdown.go index 9be01e1e9..bcd728c9d 100644 --- a/internal/output/markdown.go +++ b/internal/output/markdown.go @@ -39,6 +39,22 @@ func (m *Markdown) SetEmbedEvaluationLog(path string) { m.evaluationLog = path } +// resultEmoji maps a gemara.Result to its corresponding status emoji constant. +func resultEmoji(r gemara.Result) string { + switch r { + case gemara.Passed: + return complytime.StatusPassed + case gemara.Failed: + return complytime.StatusFailed + case gemara.NotApplicable, gemara.NotRun: + return complytime.StatusSkipped + case gemara.Unknown, gemara.NeedsReview: + return complytime.StatusError + default: + return complytime.StatusError + } +} + type summaryCounts struct { passed int failed int @@ -172,7 +188,9 @@ func (m *Markdown) writeSummary(sb *strings.Builder, now time.Time) { fmt.Fprintf(sb, "**Overall: %s -- %d%% pass rate (%d/%d applicable)**\n\n", m.evalLog.Result.String(), passRate, counts.passed, applicable) - fmt.Fprintf(sb, "| Passed | Failed | Needs Review | Unknown | N/A | Not Run | Total |\n") + fmt.Fprintf(sb, "| %s Passed | %s Failed | %s Needs Review | %s Unknown | %s N/A | %s Not Run | Total |\n", + complytime.StatusPassed, complytime.StatusFailed, complytime.StatusError, + complytime.StatusError, complytime.StatusSkipped, complytime.StatusSkipped) fmt.Fprintf(sb, "|--------|--------|--------------|---------|-----|---------|-------|\n") fmt.Fprintf(sb, "| %d | %d | %d | %d | %d | %d | %d |\n\n", counts.passed, counts.failed, counts.needsReview, @@ -185,10 +203,11 @@ func (m *Markdown) writeControlsTable(sb *strings.Builder) { fmt.Fprintf(sb, "|-----------------------|--------|---------|\n") for _, ce := range m.evalLog.Evaluations { - fmt.Fprintf(sb, "| **%s** | **%s** | %s |\n", ce.Name, ce.Result.String(), ce.Message) + fmt.Fprintf(sb, "| **%s** | **%s %s** | %s |\n", + ce.Name, resultEmoji(ce.Result), ce.Result.String(), ce.Message) for _, al := range ce.AssessmentLogs { - fmt.Fprintf(sb, "|   %s | %s | |\n", - al.Requirement.EntryId, al.Result.String()) + fmt.Fprintf(sb, "|   %s | %s %s | |\n", + al.Requirement.EntryId, resultEmoji(al.Result), al.Result.String()) } } sb.WriteString("\n") @@ -210,7 +229,7 @@ func (m *Markdown) writeFindings(sb *strings.Builder) { if firstGroup || f.result != currentResult { currentResult = f.result firstGroup = false - fmt.Fprintf(sb, "### %s\n\n", currentResult.String()) + fmt.Fprintf(sb, "### %s %s\n\n", resultEmoji(currentResult), currentResult.String()) } fmt.Fprintf(sb, "#### %s -- %s\n\n", f.requirementID, f.result.String()) diff --git a/internal/output/markdown_internal_test.go b/internal/output/markdown_internal_test.go new file mode 100644 index 000000000..098a9b943 --- /dev/null +++ b/internal/output/markdown_internal_test.go @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: Apache-2.0 + +package output + +import ( + "testing" + + "github.com/gemaraproj/go-gemara" + "github.com/stretchr/testify/assert" + + "github.com/complytime/complyctl/internal/complytime" +) + +func TestResultEmoji(t *testing.T) { + tests := []struct { + name string + result gemara.Result + want string + }{ + {"Passed", gemara.Passed, complytime.StatusPassed}, + {"Failed", gemara.Failed, complytime.StatusFailed}, + {"NotApplicable", gemara.NotApplicable, complytime.StatusSkipped}, + {"NotRun", gemara.NotRun, complytime.StatusSkipped}, + {"Unknown", gemara.Unknown, complytime.StatusError}, + {"NeedsReview", gemara.NeedsReview, complytime.StatusError}, + {"default", gemara.Result(99), complytime.StatusError}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := resultEmoji(tt.result) + assert.Equal(t, tt.want, got) + }) + } +} diff --git a/internal/output/markdown_test.go b/internal/output/markdown_test.go index d611416b9..6feaf549d 100644 --- a/internal/output/markdown_test.go +++ b/internal/output/markdown_test.go @@ -12,6 +12,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "github.com/complytime/complyctl/internal/complytime" "github.com/complytime/complyctl/internal/output" ) @@ -120,6 +121,8 @@ func TestMarkdown_Write(t *testing.T) { assert.Contains(t, content, "**ctrl-2**") assert.Contains(t, content, "req-1") assert.Contains(t, content, "req-2") + assert.Contains(t, content, complytime.StatusPassed+" Passed", + "Controls table should contain emoji status indicators") assert.Contains(t, content, "## Findings") } @@ -219,6 +222,8 @@ func TestMarkdown_FindingsWithRecommendationAndEvidence(t *testing.T) { require.NoError(t, err) content := string(data) + assert.Contains(t, content, "### "+complytime.StatusFailed+" Failed", + "findings group header should have emoji prefix") assert.Contains(t, content, "#### req-2 -- Failed") assert.Contains(t, content, "**Control**: ctrl-2") assert.Contains(t, content, "cert validity exceeds 397 days") @@ -272,10 +277,10 @@ func TestMarkdown_FindingsSortOrder(t *testing.T) { require.NoError(t, err) content := string(data) - failedIdx := strings.Index(content, "### Failed") - notApplicableIdx := strings.Index(content, "### Not Applicable") - require.Greater(t, failedIdx, 0, "expected Failed heading in findings") - require.Greater(t, notApplicableIdx, 0, "expected Not Applicable heading in findings") + failedIdx := strings.Index(content, "### "+complytime.StatusFailed+" Failed") + notApplicableIdx := strings.Index(content, "### "+complytime.StatusSkipped+" Not Applicable") + require.Greater(t, failedIdx, 0, "expected Failed heading with emoji in findings") + require.Greater(t, notApplicableIdx, 0, "expected Not Applicable heading with emoji in findings") assert.Less(t, failedIdx, notApplicableIdx, "Failed findings should appear before Not Applicable") } @@ -293,6 +298,10 @@ func TestMarkdown_PassRate(t *testing.T) { content := string(data) assert.Contains(t, content, "50% pass rate (1/2 applicable)") + assert.Contains(t, content, complytime.StatusPassed+" Passed", + "counts table header should have emoji prefix") + assert.Contains(t, content, complytime.StatusFailed+" Failed", + "counts table header should have emoji prefix") } func TestMarkdown_EmbedEvaluationLog_MissingFile(t *testing.T) { @@ -414,6 +423,8 @@ func TestMarkdown_ConfidenceLevelShown(t *testing.T) { assert.Contains(t, content, "**Confidence**: Low") assert.Contains(t, content, "**Confidence**: High") assert.NotContains(t, content, "**Confidence**: Undetermined") + assert.Contains(t, content, "### "+complytime.StatusFailed+" Failed", + "findings group header should have emoji prefix") } func TestMarkdown_ToolAttribution(t *testing.T) { @@ -471,6 +482,230 @@ func TestMarkdown_ControlsTableShowsAllControls(t *testing.T) { assert.Contains(t, content, "  req-1") assert.Contains(t, content, "  req-2") assert.Contains(t, content, "  req-3") + assert.Contains(t, content, complytime.StatusPassed+" Passed", + "passed control should have emoji prefix") + assert.Contains(t, content, complytime.StatusFailed+" Failed", + "failed control should have emoji prefix") + assert.Contains(t, content, complytime.StatusSkipped+" Not Applicable", + "not applicable control should have emoji prefix") +} + +func TestMarkdown_ControlsTableStatusIndicators(t *testing.T) { + outDir := t.TempDir() + log := &gemara.EvaluationLog{ + Metadata: gemara.Metadata{Id: "pol"}, + Result: gemara.Failed, + Evaluations: []*gemara.ControlEvaluation{ + { + Name: "ctrl-pass", + Result: gemara.Passed, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-pass"}, + Result: gemara.Passed, + Message: "ok", + }, + }, + }, + { + Name: "ctrl-fail", + Result: gemara.Failed, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-fail"}, + Result: gemara.Failed, + Message: "bad", + }, + }, + }, + { + Name: "ctrl-na", + Result: gemara.NotApplicable, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-na"}, + Result: gemara.NotApplicable, + Message: "n/a", + }, + }, + }, + { + Name: "ctrl-mixed", + Result: gemara.Failed, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-notrun"}, + Result: gemara.NotRun, + Message: "skipped", + }, + { + Requirement: gemara.EntryMapping{EntryId: "req-unknown"}, + Result: gemara.Unknown, + Message: "unknown", + }, + { + Requirement: gemara.EntryMapping{EntryId: "req-review"}, + Result: gemara.NeedsReview, + Message: "needs review", + }, + }, + }, + }, + } + md := output.NewMarkdown("pol", log) + + path, err := md.Write(outDir) + require.NoError(t, err) + + data, err := os.ReadFile(path) + require.NoError(t, err) + content := string(data) + + // Extract the Controls table section for positional specificity. + controlsStart := strings.Index(content, "## Controls") + findingsStart := strings.Index(content, "## Findings") + require.Greater(t, controlsStart, 0, "expected ## Controls section") + require.Greater(t, findingsStart, controlsStart, "expected ## Findings after ## Controls") + controlsSection := content[controlsStart:findingsStart] + + // Control rows: emoji prefixed in bold Result column + assert.Contains(t, controlsSection, "**"+complytime.StatusPassed+" Passed**", + "Passed control should display ✅ Passed in Controls table") + assert.Contains(t, controlsSection, "**"+complytime.StatusFailed+" Failed**", + "Failed control should display ❌ Failed in Controls table") + assert.Contains(t, controlsSection, "**"+complytime.StatusSkipped+" Not Applicable**", + "Not Applicable control should display ⏭️ Not Applicable in Controls table") + + // Requirement sub-rows: emoji prefixed in Result column + assert.Contains(t, controlsSection, complytime.StatusPassed+" Passed", + "Passed requirement should display ✅ Passed in Controls table") + assert.Contains(t, controlsSection, complytime.StatusFailed+" Failed", + "Failed requirement should display ❌ Failed in Controls table") + assert.Contains(t, controlsSection, complytime.StatusSkipped+" Not Applicable", + "Not Applicable requirement should display ⏭️ Not Applicable in Controls table") + assert.Contains(t, controlsSection, complytime.StatusSkipped+" Not Run", + "Not Run requirement should display ⏭️ Not Run in Controls table") + assert.Contains(t, controlsSection, complytime.StatusError+" Unknown", + "Unknown requirement should display ⚠️ Unknown in Controls table") + assert.Contains(t, controlsSection, complytime.StatusError+" Needs Review", + "Needs Review requirement should display ⚠️ Needs Review in Controls table") +} + +func TestMarkdown_SummaryCountsTableHeaders(t *testing.T) { + outDir := t.TempDir() + log := mockGemaraEvalLogWithFindings() + md := output.NewMarkdown("test-policy", log) + + path, err := md.Write(outDir) + require.NoError(t, err) + + data, err := os.ReadFile(path) + require.NoError(t, err) + content := string(data) + + // Extract the summary section (before ## Controls) for positional specificity. + controlsStart := strings.Index(content, "## Controls") + require.Greater(t, controlsStart, 0, "expected ## Controls section") + summarySection := content[:controlsStart] + + assert.Contains(t, summarySection, complytime.StatusPassed+" Passed", + "counts table header should have ✅ Passed") + assert.Contains(t, summarySection, complytime.StatusFailed+" Failed", + "counts table header should have ❌ Failed") + assert.Contains(t, summarySection, complytime.StatusError+" Needs Review", + "counts table header should have ⚠️ Needs Review") + assert.Contains(t, summarySection, complytime.StatusError+" Unknown", + "counts table header should have ⚠️ Unknown") + assert.Contains(t, summarySection, complytime.StatusSkipped+" N/A", + "counts table header should have ⏭️ N/A") + assert.Contains(t, summarySection, complytime.StatusSkipped+" Not Run", + "counts table header should have ⏭️ Not Run") + assert.Contains(t, summarySection, "| Total |", + "Total header should remain without emoji prefix") +} + +func TestMarkdown_FindingsGroupHeaderEmoji(t *testing.T) { + outDir := t.TempDir() + log := &gemara.EvaluationLog{ + Metadata: gemara.Metadata{Id: "pol"}, + Result: gemara.Failed, + Evaluations: []*gemara.ControlEvaluation{ + { + Name: "ctrl-1", + Result: gemara.Failed, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-fail"}, + Result: gemara.Failed, + Message: "failed", + }, + }, + }, + { + Name: "ctrl-2", + Result: gemara.NeedsReview, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-review"}, + Result: gemara.NeedsReview, + Message: "needs review", + }, + }, + }, + { + Name: "ctrl-3", + Result: gemara.NotApplicable, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-na"}, + Result: gemara.NotApplicable, + Message: "not applicable", + }, + }, + }, + { + Name: "ctrl-4", + Result: gemara.Unknown, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-unknown"}, + Result: gemara.Unknown, + Message: "unknown", + }, + }, + }, + { + Name: "ctrl-5", + Result: gemara.NotRun, + AssessmentLogs: []*gemara.AssessmentLog{ + { + Requirement: gemara.EntryMapping{EntryId: "req-notrun"}, + Result: gemara.NotRun, + Message: "not run", + }, + }, + }, + }, + } + md := output.NewMarkdown("pol", log) + + path, err := md.Write(outDir) + require.NoError(t, err) + + data, err := os.ReadFile(path) + require.NoError(t, err) + content := string(data) + + assert.Contains(t, content, "### "+complytime.StatusFailed+" Failed", + "Failed findings group should have ❌ emoji") + assert.Contains(t, content, "### "+complytime.StatusError+" Unknown", + "Unknown findings group should have ⚠️ emoji") + assert.Contains(t, content, "### "+complytime.StatusError+" Needs Review", + "Needs Review findings group should have ⚠️ emoji") + assert.Contains(t, content, "### "+complytime.StatusSkipped+" Not Applicable", + "Not Applicable findings group should have ⏭️ emoji") + assert.Contains(t, content, "### "+complytime.StatusSkipped+" Not Run", + "Not Run findings group should have ⏭️ emoji") } func TestMarkdown_NeedsReviewInFindings(t *testing.T) { @@ -533,8 +768,8 @@ func TestMarkdown_NeedsReviewInFindings(t *testing.T) { assert.Contains(t, content, "**ctrl-review**") assert.Contains(t, content, "Needs Review") - // Findings section should include the NeedsReview finding - assert.Contains(t, content, "### Needs Review") + // Findings section should include the NeedsReview finding with emoji + assert.Contains(t, content, "### "+complytime.StatusError+" Needs Review") assert.Contains(t, content, "#### req-review -- Needs Review") assert.Contains(t, content, "**Control**: ctrl-review") assert.Contains(t, content, "automated check inconclusive") diff --git a/openspec/changes/ci-step-summary-report/.openspec.yaml b/openspec/changes/ci-step-summary-report/.openspec.yaml new file mode 100644 index 000000000..074342d55 --- /dev/null +++ b/openspec/changes/ci-step-summary-report/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-09 diff --git a/openspec/changes/ci-step-summary-report/design.md b/openspec/changes/ci-step-summary-report/design.md new file mode 100644 index 000000000..f2804ca6b --- /dev/null +++ b/openspec/changes/ci-step-summary-report/design.md @@ -0,0 +1,111 @@ +## Context + +The Markdown report generated by `complyctl scan --format pretty` is already +well-structured on `main` with summary metadata, pass/fail counts, a controls +table, findings with collapsible evidence, and the evaluation log in a +`
` block. However, CI consumers (specifically `org-infra`'s +`reusable_compliance.yml`) still maintain a separate Python script to produce +GitHub Actions Step Summaries because the report lacks visual status indicators +for quick scanning. + +The existing codebase already defines emoji status constants in +`internal/complytime/consts.go`: +- `StatusPassed = "✅"` +- `StatusFailed = "❌"` +- `StatusSkipped = "⏭️"` +- `StatusError = "⚠️"` + +These are used in the terminal scan summary (`scan_summary.go`) and +`complyctl doctor` output but are not yet used in the Markdown report. + +## Goals / Non-Goals + +### Goals + +- Add emoji status indicators to the Controls table so each control and + requirement row shows its pass/fail status at a glance +- Add emoji to the summary counts table headers for scanability +- Ensure the report renders correctly when used as a GitHub Actions Step + Summary (`cat report-*.md >> $GITHUB_STEP_SUMMARY`) +- Maintain backward compatibility -- the report is still a valid standalone + Markdown file + +### Non-Goals + +- Changing the report structure or adding/removing sections +- Adding a `--ci` or `--step-summary` flag (the same report serves both uses) +- Modifying the terminal scan summary output (already has emoji) +- Changing the evaluation log embedding behavior +- Upstream changes to `org-infra` (separate follow-up) + +## Decisions + +### D1: Reuse existing `complytime.Status*` constants + +**Decision**: Import and use the existing `StatusPassed`, `StatusFailed`, +`StatusSkipped`, and `StatusError` constants from `internal/complytime/consts.go`. + +**Rationale**: These constants are already used across the codebase for +consistent emoji representation. Using the same constants ensures visual +consistency between the terminal summary and the Markdown report. + +**Alternative considered**: Defining Markdown-specific text labels +(e.g., "PASS", "FAIL") instead of emoji. Rejected because emoji render +well in GitHub Markdown and match the existing CI Python script's behavior. + +### D2: Add emoji as a prefix to the Result column in Controls table + +**Decision**: Prepend the emoji to the Result column value in +`writeControlsTable()`, so each row reads e.g., `✅ Passed` or `❌ Failed`. + +**Rationale**: Keeps the table compact while providing both visual indicator +and text label. The emoji appears before the text so the eye catches the +status pattern when scanning vertically. + +**Alternative considered**: Adding a separate "Status" column with only emoji. +Rejected because it widens the table without adding information beyond what +the Result column already provides. + +### D3: Map `gemara.Result` to status emoji via a helper function + +**Decision**: Create a `resultEmoji(r gemara.Result) string` helper in +`markdown.go` that maps each result type to its emoji constant. This +centralizes the mapping and keeps `writeControlsTable` and `writeSummary` +clean. + +**Rationale**: The mapping is the same one used in `scan_summary.go` but +operates on `gemara.Result` rather than aggregated provider results. A +dedicated helper avoids duplicating the switch statement. + +### D4: Add emoji to summary counts table headers + +**Decision**: Prefix each column header in the counts table with its +corresponding emoji (e.g., `✅ Passed | ❌ Failed | ...`). + +**Rationale**: Makes the summary row scannable at a glance -- the user can +see the overall distribution without reading the text. This matches the +visual language of GitHub Actions Step Summaries in other compliance tools. + +### D5: Add emoji to findings group headers + +**Decision**: Prefix the findings group headers (e.g., `### Failed`) with +the corresponding emoji (e.g., `### ❌ Failed`). + +**Rationale**: When scrolling through findings, the emoji makes group +boundaries visually distinct. This is especially useful in long reports. + +## Risks / Trade-offs + +- **[Risk] Emoji rendering in non-GitHub contexts** → The report is primarily + consumed as a GitHub Step Summary. Emoji render correctly in all modern + Markdown renderers. Plain text fallback (the text label is always present) + ensures readability even if emoji don't render. + +- **[Risk] GitHub Step Summary size limits** → GitHub truncates Step Summaries + at ~1 MB. Adding emoji to each row increases size by ~4 bytes per row, + which is negligible. The evaluation log `
` block is the largest + contributor and is already collapsible. + +- **[Risk] Test maintenance** → Existing tests assert on exact Markdown output + strings. Adding emoji changes these strings. Mitigation: update existing + assertions and add new tests specifically for emoji presence. diff --git a/openspec/changes/ci-step-summary-report/proposal.md b/openspec/changes/ci-step-summary-report/proposal.md new file mode 100644 index 000000000..75da6bdb0 --- /dev/null +++ b/openspec/changes/ci-step-summary-report/proposal.md @@ -0,0 +1,83 @@ +## Why + +CI workflows that consume `complyctl scan --format pretty` currently duplicate +the report logic with custom Python scripts to produce a GitHub Actions Step +Summary (see `org-infra/reusable_compliance.yml`). This creates two divergent +representations of the same compliance data, doubling the maintenance surface +and requiring a `pip install pyyaml` dependency in CI. + +The `main` branch already has significant improvements (summary metadata table, +pass/fail counts, controls table, findings with collapsible evidence, evaluation +log in `
`). Two remaining gaps prevent the Markdown report from being +directly usable as a Step Summary: + +1. **No status indicators** -- the Controls table lacks visual pass/fail emoji, + forcing users to read every row to find failures. +2. **Step Summary rendering verification** -- `
/` blocks, + Markdown tables, and report length have not been validated against GitHub's + Step Summary renderer and its ~1 MB size limit. + +Closing these gaps lets CI consumers replace ~55 lines of custom Python with +a simple `cat report-*.md >> $GITHUB_STEP_SUMMARY`. + +## What Changes + +- Add emoji status indicators to the Controls table in the Markdown report + (reusing existing `complytime.Status*` constants: `StatusPassed`, + `StatusFailed`, `StatusSkipped`, `StatusError`) +- Add emoji status indicators to the summary counts table for scanability +- Update existing unit tests and add new tests to verify the indicators render + correctly +- Verify `
/` blocks and Markdown tables are compatible with + GitHub Actions Step Summary rendering constraints + +## Capabilities + +### New Capabilities + +- `ci-scannable-report`: Add emoji status indicators to the Markdown report's + Controls table and summary section so the report is scannable at a glance + when used as a GitHub Actions Step Summary + +### Modified Capabilities + +_(none -- no existing spec-level requirements are changing)_ + +## Impact + +- **Code**: `internal/output/markdown.go` -- `writeControlsTable()` and + `writeSummary()` gain emoji indicators; `writeFindings()` gains + result-specific emoji in group headers +- **Tests**: `internal/output/markdown_test.go` -- existing tests updated + to expect emoji in output; new tests for indicator rendering +- **Dependencies**: None (reuses existing `complytime.Status*` constants) +- **Downstream**: Enables `complytime/org-infra` to simplify + `reusable_compliance.yml` by removing custom Python summary generation + and `pip install pyyaml` dependency +- **API**: No changes to CLI flags, protobuf, or provider interface + +## Constitution Alignment + +### I. Autonomous Collaboration + +Not directly applicable -- no artifact-based communication changes. The change +is a presentational enhancement to an existing output format. + +### II. Composability First + +PASS -- the new `resultEmoji()` helper is a pure function that centralizes the +`gemara.Result` → emoji mapping. It is composable and reusable by any future +report format that needs status indicators. + +### III. Observable Quality + +PASS -- emoji status indicators improve visual scanability of compliance +reports, making quality signals (pass/fail) immediately observable without +reading every row. This directly supports the principle of making quality +claims visible and actionable. + +### IV. Testability + +PASS -- each emoji mapping is independently testable via a table-driven unit +test. Every insertion point (controls table, summary headers, findings headers) +is verified through dedicated test cases. diff --git a/openspec/changes/ci-step-summary-report/specs/ci-scannable-report/spec.md b/openspec/changes/ci-step-summary-report/specs/ci-scannable-report/spec.md new file mode 100644 index 000000000..e64836202 --- /dev/null +++ b/openspec/changes/ci-step-summary-report/specs/ci-scannable-report/spec.md @@ -0,0 +1,130 @@ +## ADDED Requirements + +### Requirement: Controls table shows status indicators + +The Markdown report's Controls table MUST display an emoji status indicator +as a prefix to the Result column value for each control row and each +requirement row. The indicator MUST use the same emoji constants as the +terminal scan summary (`StatusPassed`, `StatusFailed`, `StatusSkipped`, +`StatusError`). + +#### Scenario: Passed control displays check mark + +- **GIVEN** a completed compliance scan with evaluation results +- **WHEN** a control evaluation has result `Passed` +- **THEN** the Controls table row for that control MUST display `✅ Passed` in the Result column + +#### Scenario: Failed control displays cross mark + +- **GIVEN** a completed compliance scan with evaluation results +- **WHEN** a control evaluation has result `Failed` +- **THEN** the Controls table row for that control MUST display `❌ Failed` in the Result column + +#### Scenario: Not Applicable control displays skip indicator + +- **GIVEN** a completed compliance scan with evaluation results +- **WHEN** a control evaluation has result `Not Applicable` +- **THEN** the Controls table row for that control MUST display `⏭️ Not Applicable` in the Result column + +#### Scenario: Not Run result displays skip indicator + +- **GIVEN** a completed compliance scan with evaluation results +- **WHEN** an assessment log has result `Not Run` +- **THEN** the requirement row MUST display `⏭️ Not Run` in the Result column + +#### Scenario: Unknown result displays warning indicator + +- **GIVEN** a completed compliance scan with evaluation results +- **WHEN** an assessment log has result `Unknown` +- **THEN** the requirement row MUST display `⚠️ Unknown` in the Result column + +#### Scenario: Needs Review result displays warning indicator + +- **GIVEN** a completed compliance scan with evaluation results +- **WHEN** an assessment log has result `Needs Review` +- **THEN** the requirement row MUST display `⚠️ Needs Review` in the Result column + +#### Scenario: Requirement rows show individual status indicators + +- **GIVEN** a completed compliance scan with evaluation results +- **WHEN** a control has multiple assessment logs with mixed results +- **THEN** each requirement row MUST display its own emoji prefix matching its individual result + +### Requirement: Summary counts table shows status indicators + +The summary counts table headers MUST be prefixed with their corresponding +emoji so the distribution is scannable at a glance without reading text. + +#### Scenario: Counts table headers include emoji + +- **GIVEN** a Markdown report is being generated for a policy scan +- **WHEN** the summary counts table is rendered +- **THEN** the counts table header row MUST display `✅ Passed | ❌ Failed | ⚠️ Needs Review | ⚠️ Unknown | ⏭️ N/A | ⏭️ Not Run | Total` + +### Requirement: Findings group headers show status indicators + +The findings section group headers MUST be prefixed with their corresponding +emoji to make group boundaries visually distinct when scrolling. + +#### Scenario: Failed findings group header + +- **GIVEN** a completed compliance scan with non-passing results +- **WHEN** the findings section contains a group of Failed findings +- **THEN** the group header MUST display `### ❌ Failed` + +#### Scenario: Needs Review findings group header + +- **GIVEN** a completed compliance scan with non-passing results +- **WHEN** the findings section contains a group of Needs Review findings +- **THEN** the group header MUST display `### ⚠️ Needs Review` + +#### Scenario: Not Applicable findings group header + +- **GIVEN** a completed compliance scan with non-passing results +- **WHEN** the findings section contains a group of Not Applicable findings +- **THEN** the group header MUST display `### ⏭️ Not Applicable` + +#### Scenario: Unknown findings group header + +- **GIVEN** a completed compliance scan with non-passing results +- **WHEN** the findings section contains a group of Unknown findings +- **THEN** the group header MUST display `### ⚠️ Unknown` + +#### Scenario: Not Run findings group header + +- **GIVEN** a completed compliance scan with non-passing results +- **WHEN** the findings section contains a group of Not Run findings +- **THEN** the group header MUST display `### ⏭️ Not Run` + +### Requirement: Report is structurally compatible with GitHub Actions Step Summary + +The Markdown report MUST produce valid HTML/Markdown structures that are +compatible with GitHub Actions Step Summary rendering when appended via +`cat report-*.md >> $GITHUB_STEP_SUMMARY`. + +#### Scenario: Details blocks use valid HTML structure + +- **GIVEN** a Markdown report with an embedded evaluation log +- **WHEN** the report contains `
` blocks +- **THEN** every `
` tag MUST have a matching `
` closing tag +- **AND** every `
` block MUST contain a `` element + +#### Scenario: Tables have consistent column counts + +- **GIVEN** a Markdown report with tables +- **WHEN** the report contains Markdown tables +- **THEN** all table rows MUST have the same number of columns as the header row + +#### Scenario: Report stays within size limits + +- **GIVEN** a policy scan with up to 200 controls +- **WHEN** the Markdown report is generated +- **THEN** the report size MUST be less than 512 KB + +## MODIFIED Requirements + +None. + +## REMOVED Requirements + +None. diff --git a/openspec/changes/ci-step-summary-report/tasks.md b/openspec/changes/ci-step-summary-report/tasks.md new file mode 100644 index 000000000..4eceefdd0 --- /dev/null +++ b/openspec/changes/ci-step-summary-report/tasks.md @@ -0,0 +1,45 @@ +## 1. Result-to-Emoji Helper + +- [x] 1.1 Add `resultEmoji(r gemara.Result) string` helper function in `internal/output/markdown.go` that maps each `gemara.Result` to its `complytime.Status*` emoji constant (`Passed`→`StatusPassed`, `Failed`→`StatusFailed`, `NotApplicable`/`NotRun`→`StatusSkipped`, `Unknown`/`NeedsReview`→`StatusError`) + +## 2. Controls Table Indicators + +- [x] 2.1 Update `writeControlsTable()` in `internal/output/markdown.go` to prefix each control row's Result column with `resultEmoji(ce.Result)` (e.g., `✅ Passed`) +- [x] 2.2 Update `writeControlsTable()` requirement sub-rows to prefix Result column with `resultEmoji(al.Result)` + +## 3. Summary Counts Indicators + +- [x] 3.1 Update `writeSummary()` in `internal/output/markdown.go` to prefix each counts table header with its emoji (e.g., `✅ Passed | ❌ Failed | ⚠️ Needs Review | ⚠️ Unknown | ⏭️ N/A | ⏭️ Not Run | Total`) + +## 4. Findings Group Header Indicators + +- [x] 4.1 Update `writeFindings()` in `internal/output/markdown.go` to prefix each result group heading with `resultEmoji(currentResult)` (e.g., `### ❌ Failed`) + +## 5. Update Existing Tests + +- [x] 5.1 Update `TestMarkdown_Write` assertions in `internal/output/markdown_test.go` to expect emoji prefixes in Controls table output +- [x] 5.2 Update `TestMarkdown_ControlsTableShowsAllControls` assertions to expect emoji in Result column +- [x] 5.3 Update `TestMarkdown_PassRate` assertions for emoji in counts table headers +- [x] 5.4 Update `TestMarkdown_FindingsSortOrder` assertions for emoji in findings group headers +- [x] 5.5 Update `TestMarkdown_ConfidenceLevelShown` assertions for emoji in findings headers +- [x] 5.6 Update `TestMarkdown_FindingsWithRecommendationAndEvidence` assertions for emoji in findings headers + +## 6. Add New Tests + +- [x] 6.1 Add table-driven `TestResultEmoji` unit test exercising all 6 `gemara.Result` types plus a default case +- [x] 6.2 Add `TestMarkdown_ControlsTableStatusIndicators` test verifying each result type renders with the correct emoji prefix in the Controls table +- [x] 6.3 Add `TestMarkdown_SummaryCountsTableHeaders` test verifying counts table headers contain emoji prefixes +- [x] 6.4 Add `TestMarkdown_FindingsGroupHeaderEmoji` test verifying findings group headers contain emoji prefixes for all non-Passed result types + +## 7. Verification + +- [x] 7.1 Run `make test-unit` and confirm all tests pass +- [x] 7.2 Run `make lint` and confirm no lint violations +- [x] 7.3 Verify a sample report: generate report output and confirm (a) emoji render as graphical icons, (b) `
` blocks have matching open/close tags, (c) table column counts are consistent across rows, (d) no raw HTML artifacts appear in rendered view + +## 8. Documentation + +- [x] 8.1 Update `CHANGELOG.md` with entry for emoji status indicators in Markdown report +- [x] 8.2 Update `AGENTS.md` Recent Changes section with `ci-step-summary-report` entry + +