ci: add mutation testing with mutant (informational, license-gated) - #44
Open
Amayyas wants to merge 2 commits into
Open
ci: add mutation testing with mutant (informational, license-gated)#44Amayyas wants to merge 2 commits into
Amayyas wants to merge 2 commits into
Conversation
- mutant + mutant-rspec added, require: false (not autoloaded) - .mutant.yml scopes subjects to Chorus::Router, Chorus::Context, and Chorus::Orchestrator — the classes with real conditional logic and behavioral specs. Deliberately excludes Chorus::Client (every spec mocks it entirely, see #9 — mutating unexercised HTTP code would just be noise) and the thin agent wrapper classes. - .github/workflows/mutation.yml: continue-on-error, and NOT wired to push/pull_request yet — mutant's free Open Source License requires registering this repo under a separate account (https://github.com/mbj/mutant), which isn't automatable. Triggers are workflow_dispatch + weekly schedule for now; once registered and the runtime on this codebase is known, this can move to push/pull_request. - CONTRIBUTING.md documents how to run it locally and the registration requirement. Neither this nor local testing could fully verify the mutant run end-to-end: mutant depends on parser/racc, which needs native compilation unavailable in this dev environment, and the license gate blocks a real CI run until the repo is registered. Config was written carefully against mutant's documented schema, but the first real verification will be whoever runs it after registering. Closes #26
📝 WalkthroughWalkthroughThe project adds Mutant dependencies and configuration for three Chorus classes. GitHub Actions runs mutation tests manually or every Monday. Contributor documentation describes local setup, licensing, and the non-blocking CI job. ChangesMutation testing
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
mutant/mutant-rspec 0.16.0 raised their required Ruby version to >= 3.3, which broke bundle install on every job pinned to Ruby 3.2 (Tests 3.2, RuboCop, bundler-audit, Gem builds) — verified in CI on this same PR. 0.15.1 is the latest release still supporting 3.2, matching this project's required_ruby_version and CI matrix.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/mutation.yml:
- Line 20: Update the workflow permissions to grant only contents: read, and
configure the actions/checkout step to use persist-credentials: false. Apply
both changes to the checkout workflow around the existing actions/checkout@v5
step so the later bundle exec mutant command cannot inherit a writable token.
In `@CONTRIBUTING.md`:
- Line 41: Update the contributor-facing wording in CONTRIBUTING.md to hyphenate
“open source” as “open-source” when it functions as a compound adjective,
without changing the surrounding license instruction.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 46f3c4c5-f006-4e90-a7cd-768f453da899
📒 Files selected for processing (4)
.github/workflows/mutation.yml.mutant.ymlCONTRIBUTING.mdGemfile
| runs-on: ubuntu-latest | ||
| continue-on-error: true | ||
| steps: | ||
| - uses: actions/checkout@v5 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## workflow files"
git ls-files '.github/workflows/*' || true
echo "## mutation workflow"
if [ -f .github/workflows/mutation.yml ]; then
nl -ba .github/workflows/mutation.yml | sed -n '1,180p'
fi
echo "## Gemfile indicators in mutation workflow"
rg -n "persist-credentials|permissions:|actions/checkout|bundler|bundle install|Gemfile|git |GH_TOKEN|GITHUB_TOKEN|checkout@v5|workflow_dispatch|schedule" .github/workflows/mutation.yml || true
echo "## repo Gemfiles (read-only)"
git ls-files | grep -E '(^|/)Gemfile$|\.gemspec$' || true
echo "## Gemfile git dependency occurrences"
rg -n "source.*https:/|^ *git [\"']|git\s" -g 'Gemfile*' -g '*.gemspec' . || trueRepository: Amayyas/Chorus
Length of output: 378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## workflows"
git ls-files '.github/workflows/*' || true
echo "## mutation workflow"
if [ -f .github/workflows/mutation.yml ]; then
nl -ba .github/workflows/mutation.yml | sed -n '1,180p'
fi
echo "## relevant workflow options"
rg -nE "persist-credentials|permissions:|actions/checkout|bundler|bundle install|Gemfile|git |GH_TOKEN|GITHUB_TOKEN|workflow_dispatch|schedule" .github/workflows/mutation.yml || true
echo "## repo Gemfiles"
git ls-files | grep -E '(^|/)Gemfile$|\.gemspec$' || true
echo "## Gemfile gemspec git dependency occurrences"
rg -nE "source .*https:/|^ *git [\"']|git\s" -g 'Gemfile*' -g '*.gemspec' . || trueRepository: Amayyas/Chorus
Length of output: 373
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## workflow files"
git ls-files '.github/workflows/*'
echo "## mutation workflow"
cat -n .github/workflows/mutation.yml
echo "## relevant workflow options"
grep -nE 'persist-credentials|permissions:|actions/checkout|bundler|bundle install|Gemfile|git |GH_TOKEN|GITHUB_TOKEN|workflow_dispatch|schedule' .github/workflows/mutation.yml || true
echo "## repo Gemfiles"
git ls-files | grep -E '(^|/)Gemfile$|\.gemspec$' || true
echo "## Gemfile gemspec git dependency occurrences"
grep -RInE "source .*https:/|^ *git [\"']|git\s" -g 'Gemfile*' -g '*.gemspec' . || trueRepository: Amayyas/Chorus
Length of output: 1789
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## Gemfile"
cat -n Gemfile
echo "## Gemspec"
cat -n chorus-llm.gemspec
echo "## actions checkout documentation option"
python3 - <<'PY'
import urllib.request
urls = [
"https://raw.githubusercontent.com/actions/checkout/main/README.md",
]
for url in urls:
try:
data = urllib.request.urlopen(url, timeout=20).read().decode("utf-8", "replace")
except Exception as exc:
print(f"FETCH_ERROR {url}: {exc}")
continue
for needle in ["persist-credentials", "token", "git config --local --unset"]:
print(f"\n## occurrences for {needle!r} in {url}")
for i, line in enumerate(data.splitlines(), 1):
if needle in line:
print(f"{i}: {line}")
break
PYRepository: Amayyas/Chorus
Length of output: 2386
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from urllib.request import urlopen, Request
url = "https://raw.githubusercontent.com/actions/checkout/main/README.md"
req = Request(url, headers={"User-Agent": "CodeRabbit-review/1.0"})
data = urlopen(req, timeout=20).read().decode("utf-8", "replace")
lines = data.splitlines()
print("## actions/checkout persist-credentials evidence")
for i, line in enumerate(lines, 1):
if "persist-credentials" in line or "credential" in line.lower() or "git config --local --unset" in line:
print(f"{i}: {line}")
PY
echo "## Gemfile git dependency occurrences"
grep -RInE "source .*https:/|^ *git [\"']|git\s" --include='Gemfile*' --include='*.gemspec' . || trueRepository: Amayyas/Chorus
Length of output: 2438
Disable checkout credential persistence.
This workflow only needs to read the repo contents and then execute bundle exec mutant; the Gemfile and gemspec use RubyGems sources with no Git dependencies. Set permissions: contents: read and configure checkout with persist-credentials: false so the later script does not inherit an actionable token by default.
🧰 Tools
🪛 zizmor (1.28.0)
[warning] 20-20: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/mutation.yml at line 20, Update the workflow permissions
to grant only contents: read, and configure the actions/checkout step to use
persist-credentials: false. Apply both changes to the checkout workflow around
the existing actions/checkout@v5 step so the later bundle exec mutant command
cannot inherit a writable token.
Source: Linters/SAST tools
| Runs [mutant](https://github.com/mbj/mutant) against the classes with real | ||
| conditional logic (`Chorus::Router`, `Chorus::Context`, `Chorus::Orchestrator` | ||
| — see `.mutant.yml`) to check whether the specs would actually catch a | ||
| regression, not just that they pass. Free for public open source use, but |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Hyphenate open-source.
Use the compound adjective in this contributor-facing license instruction.
Proposed wording fix
-Free for public open source use, but requires registering this repo once under mutant's Free Project License —
+Free for public open-source use, but requires registering this repo once under mutant's Free Project License —📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| regression, not just that they pass. Free for public open source use, but | |
| regression, not just that they pass. Free for public open-source use, but |
🧰 Tools
🪛 LanguageTool
[grammar] ~41-~41: Use a hyphen to join words.
Context: ...ust that they pass. Free for public open source use, but requires registering thi...
(QB_NEW_EN_HYPHEN)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@CONTRIBUTING.md` at line 41, Update the contributor-facing wording in
CONTRIBUTING.md to hyphenate “open source” as “open-source” when it functions as
a compound adjective, without changing the surrounding license instruction.
Source: Linters/SAST tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mutant+mutant-rspecadded (require: false).mutant.ymlscopes subjects toChorus::Router,Chorus::Context,Chorus::Orchestrator— the classes with real conditional logic. Deliberately excludesChorus::Client(fully mocked in specs, see No HTTP-contract-level test for Client's request shape #9) and the thin agent classes..github/workflows/mutation.yml:continue-on-error, triggers areworkflow_dispatch+ weekly schedule only (not push/PR yet)CONTRIBUTING.mddocuments local usage + the registration requirementCloses #26
Why this needs a follow-up from you
mutant is free for public OSS use, but requires registering this repo under mutant's Free Project License at a separate account (https://github.com/mbj/mutant) — not something I can do on your behalf. Until that's done,
--usage opensourcewill fail its license check. The workflow iscontinue-on-errorso this won't block anything either way.Honesty about verification
I could not fully verify this end-to-end:
parser/racc, which needs native compilation unavailable in this dev environment (same root cause as not being able to run RuboCop locally all along).The config was written carefully against mutant's documented schema (verified via the actual README/docs, not guessed), but the first real confirmation that it works will be whoever runs it after registering the repo.
Test plan
.mutant.yml, workflow file)bundle exec rspec— 34/34 passing (mutant isn't autoloaded, no interference)workflow_dispatchthis workflow to confirm it actually worksSummary by CodeRabbit
Tests
Documentation