Run every native test CI builds, and fail when one cannot be built - #160
Conversation
|
Warning Review limit reached
Next review available in: 50 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
WalkthroughThe native test setup now supports vendored or system cJSON. CI enables all tests, discovers every built test executable, records failures, and fails the job when any test fails. ChangesNative test infrastructure
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Line 150: Fix the unmatched double quote in the native-test step’s echo
command so the “All native tests passed.” message is a complete shell string and
the workflow can parse and execute the build and tests.
- Around line 127-150: Add a job-level permissions block to the native-tests job
with contents set to read, without granting additional scopes unless an existing
command in that job explicitly requires them. Keep the current test execution
steps unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 68a19eaa-1096-4a8c-b3ee-4a6f48ab5844
📒 Files selected for processing (2)
.github/workflows/ci.ymltest/native/CMakeLists.txt
1 participant
The native test job compiled 14 test binaries and executed 5. Three more were never built at all. Every check was green throughout.
What was not running
test_protocol,test_coordinatorandtest_coordinator_raceare gated on${CJSON_DIR}/cJSON.c, which lives undermanaged_components/. That directory is generated by the IDF component manager and excluded by.gitignore, and this job never runs an IDF build, so the path is absent on a fresh checkout. CMake configured those targets away with a STATUS message, and the job then guarded the run with[ ! -f ./test_protocol ] || ./test_protocol, so a binary that was never built read as a pass. The CI log shows it plainly:A further seven were compiled every run and never executed, because the list of tests to run was maintained by hand and had drifted from the list of tests that exist:
test_anti_glitch,test_hw_entropy,test_hw_entropy_sha,test_integration,test_psbt_fraud,test_secresult,test_self_test.test_protocolcovers protocol parsing, which is a trust boundary on this device.test_coordinator_raceis a ThreadSanitizer race test.test_hw_entropyandtest_self_testcover the entropy path behind the v0.2.1 fix.Changes
cJSON discovery now prefers the vendored copy when a build has produced it, so tests compile the same cJSON as the firmware, and otherwise falls back to a system cJSON, which is what a bare checkout has.
REQUIRE_ALL_TESTSturns "not found" into a configure error; CI sets it, so a dependency that quietly disappears fails the job instead of taking its tests with it. Local developers can still configure without it.The job now runs every executable it built rather than a fixed list, so a new test is picked up by existing it, and collects failures instead of stopping at the first.
Verification
Both discovery paths were exercised. With
managed_components/present, configuration reportscJSON: vendored. With it moved aside, reproducing a fresh checkout, it reportscJSON: systemand all four previously-skipped targets build and pass.The full suite was built and run locally before changing the job, to confirm that enabling seven previously-unexecuted binaries would not turn CI red: 18 built, 18 passed, 0 failed.
Summary by CodeRabbit
Tests
Chores