CI/CD Pipeline Evolution: Security Hardening, Sharding, & Performance Budgets

[fderuiter]

CI/CD Pipeline Evolution: Security Hardening, Sharding, & Performance Budgets

• Type: DevOps / CI/CD
• Description: As this tool transitions into a comprehensive clinical research utility handling complex logic and generating "Certificates of Randomization," our CI/CD pipeline must enforce strict quality, security, and performance gates. The current GitHub Actions workflows (ci.yml, deploy.yml) and Playwright/Vitest setups need to be scaled so that developer velocity doesn't drop as test volume explodes.
• Proposed Solution:
  • Playwright Sharding & Parallelization: As we add tests for 18+ complex UI features (Analytics, Wizards, Bulk Imports), E2E test execution time will skyrocket. Update the .github/workflows/ci.yml to shard Playwright tests across multiple parallel GitHub Action runners to enforce a strict < 5 minute pipeline execution time.
  • Security Scanning (SAST): Integrate CodeQL or a similar Static Application Security Testing tool into the PR pipeline. Because we are introducing cryptographic audit trails and dealing with randomization integrity, the pipeline must automatically fail if weak PRNGs (like unseeded Math.random) or vulnerable crypto implementations are detected in the algorithmic domain.
  • Bundle Size & Performance Budgets: With the planned addition of export libraries (like xlsx), charting tools for analytics, and crypto utilities, the application payload will bloat. Implement strict Angular bundle size budgets in angular.json and enforce them in CI to prevent regressions in Time-to-Interactive.
  • Automated Accessibility (a11y) Audits: Clinical tooling often requires strict accessibility compliance. Integrate Lighthouse CI or axe-core into the Playwright pipeline to automatically fail PRs that introduce components violating WCAG 2.1 AA standards.

[fderuiter]

Phase 1: Test Infrastructure Scaling

Ticket 20.A: Implement Playwright Test Sharding and CI Parallelization

Type: DevOps / Test Infrastructure
Context: As we add complex UI features (multi-step wizards, bulk imports), our Playwright E2E test suite will grow exponentially. The current .github/workflows/ci.yml runs tests sequentially on a single runner. We must parallelize this execution to maintain a rapid feedback loop for developers.
Implementation Steps:

1. Update .github/workflows/ci.yml to utilize GitHub Actions matrix strategy to spin up multiple parallel runners (e.g., shard: [1, 2, 3, 4]).
2. Modify the Playwright execution command in the workflow to accept the shard arguments: npx playwright test --shard=${{ matrix.shard }}/${{ strategy.job-total }}.
3. Add a new dependent job in the workflow to download the blob test artifacts from all parallel shards and merge them into a single, unified HTML test report using npx playwright merge-reports.
4. Upload the final merged HTML report as a workflow artifact.
   Definition of Done (DoD):

• E2E tests execute across at least 3 parallel runners.
• Total E2E pipeline execution time is strictly under 5 minutes.
• A single, unified test report is generated and accessible on the GitHub Actions run summary.

---

Phase 2: Security & Integrity

Ticket 20.B: Integrate Static Application Security Testing (SAST) with CodeQL

Type: DevOps / Security
Context: This application generates clinical trial schemas. The mathematical integrity of the randomization is critical. We must ensure that developers do not accidentally introduce weak Pseudo-Random Number Generators (PRNGs) or vulnerable cryptographic implementations.
Implementation Steps:

1. Create a new GitHub Actions workflow: .github/workflows/codeql.yml.
2. Configure the action to run on push to main and on pull_request.
3. Set up the CodeQL initialization for javascript and typescript.
4. Implement a custom CodeQL query (or ESLint security plugin equivalent) specifically designed to flag any usage of standard Math.random() anywhere within the domain/randomization-engine directory, enforcing the use of the seeded generator or the Web Crypto API.
   Definition of Done (DoD):

• CodeQL workflow executes successfully on PRs.
• The pipeline automatically fails if Math.random() or other insecure cryptographic primitives are detected in the algorithmic domain.
• Vulnerabilities are logged in the repository's GitHub Security tab.

---

Phase 3: Performance Guardrails

Ticket 20.C: Enforce Strict Performance Budgets and Bundle Analysis

Type: DevOps / Performance
Context: The addition of heavy libraries like xlsx, charting tools, and cryptographic utilities threatens to bloat the Angular application payload, degrading the Time-to-Interactive (TTI) for users on slower networks. We must catch bundle bloat before it merges to main.
Implementation Steps:

1. Update angular.json to define strict maximum limits for initial and anyComponentStyle budgets in the production build configuration.
2. Set the error threshold tightly around the current baseline (e.g., fail the build if the initial bundle exceeds current baseline + 100kb).
3. Integrate a bundle analysis tool (like source-map-explorer or Angular's built-in esbuild analyzer) into .github/workflows/ci.yml.
4. Add a step to the CI pipeline that comments on the Pull Request with a breakdown of the bundle size diff if the budget threshold is approached.
   Definition of Done (DoD):

• ng build fails in the CI pipeline if the defined performance budgets are exceeded.
• Developers can easily view a visual treemap of what dependencies are causing the bundle bloat via pipeline artifacts.

---

Phase 4: Compliance & Accessibility

Ticket 20.D: Automate WCAG 2.1 AA Accessibility Audits in the E2E Pipeline

Type: DevOps / QA
Context: Clinical tools often require strict accessibility compliance. Manual testing is insufficient for catching missing ARIA attributes, poor color contrast, or non-navigable DOM structures introduced during rapid feature development.
Implementation Steps:

1. Install @axe-core/playwright as a development dependency.
2. Create a custom Playwright fixture or utility function (e.g., checkA11y()) that executes the axe-core engine against the current page state.
3. Update existing E2E tests in tests_e2e/ to invoke this assertion after rendering major UI components (e.g., the configuration wizard, the results grid).
4. Configure the axe-core runner to strictly enforce WCAG 2.1 AA standards, failing the test immediately upon encountering any "critical" or "serious" violations.
   Definition of Done (DoD):

• Playwright E2E tests fail if a PR introduces an accessibility violation.
• The test output clearly identifies the offending DOM element and the specific WCAG rule violated.

[m13v]

the phased approach here is solid. on the playwright sharding, one thing that's easy to miss is that the merge-reports step needs all shards to upload their blob artifacts to the same directory structure. if a shard fails and doesn't upload, the merge silently produces an incomplete report. worth adding a check that all expected shard artifacts exist before merging. for axe-core integration, it's simpler than most people expect. literally just await new AxeBuilder({ page }).analyze() and assert on violations.length. the trick is running it after the page is fully interactive, not just after load, since a lot of a11y violations come from dynamically rendered components that aren't in the initial DOM.

[m13v]

we've been dealing with playwright sharding and test execution at scale, here's how we handle the browser orchestration: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/browser.ts - the a11y integration point would go right after the page interaction steps in a similar setup

[fderuiter]

Hey @m13v, thanks so much for the insights! I am building this solo to fill a specific industry need and pushing to wrap it up by the weekend, so this is incredibly helpful.

Quick note: the link to the browser.ts file you shared gave me a 404. Is that repo private, or do I need specific permissions to view it?

A few quick questions:

1. For the sharding merge check, do you just bash assert the blob count against the job total, or is there a native Playwright method?

2. How do you handle single shard retries without breaking that final merge?

3. For axe core, do you rely on networkidle or specific locators to ensure interactivity before running the check?

4. In your orchestration setup, are the a11y checks automatically baked into the hooks under the hood, or do they still need to be called explicitly in the spec files?

Thanks again!

[m13v]

sorry about the 404, that file is in a private repo. for the sharding merge check, yeah we just do a bash assertion comparing the blob count against expected shard total before running merge-reports. nothing fancy, just a simple count of artifact files vs expected shards and fail if they don't match.

for singleton browser instances across parallel shards, each shard gets its own browser context, not a shared instance. Playwright's test runner handles that automatically when you use --shard=N/M. keeps things isolated so you don't get state leaking between shards. what's your sharding setup look like right now?

[fderuiter]

Ah, got it on the private repo. Good to know the bash assertion is the standard approach. I will definitely add that.

As for my sharding setup, I have a working Playwright pipeline in GitHub Actions but it runs sequentially on a single runner.
It is becoming a major bottleneck.

My goal for this weekend is to refactor it into matrix strategy using the shard flag to run everything in parallel.

Appreciate you confirming how Playwright handles the isolated contexts.

Really appreciate the help!

[m13v]

you'll love the speedup. one tip for the matrix sharding setup: use the --shard flag with the total count matching your matrix size (e.g. --shard=1/4, --shard=2/4, etc.) and make sure you collect all the blob-report artifacts before running merge-reports. the merge step fails silently if a shard artifact is missing, so adding a quick check that all expected shards uploaded is worth it. good luck this weekend!