feat(enterprise): fit governed calibration reports - #546
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe change adds governed enterprise calibration-report fitting, public exports, deterministic validation, provenance controls, privacy tests, convergence-aware trace validation, and bounded HTML output publication. ChangesEnterprise calibration reporting
Essay report trace and publication controls
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant EnterpriseReporting
participant BundleBuilder
participant SharedFacetFitter
participant ReportTuple
Caller->>EnterpriseReporting: submit executions and report parameters
EnterpriseReporting->>BundleBuilder: validate and build governed bundle
EnterpriseReporting->>SharedFacetFitter: fit each criterion design
SharedFacetFitter-->>EnterpriseReporting: return shared calibration report
EnterpriseReporting-->>ReportTuple: return reports in criterion order
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
@coderabbitai review @opencode-agent @cwl-noema-review Review the exact current implementation head after the focused validation commit lands. Verify that the enterprise workflow calls the governed bundle assembler once, normalizes review triggers once, delegates each canonical criterion design once to the exact shared Rust-backed report helper, returns only |
|
|
|
@coderabbitai review Please review exact current head |
|
|
|
@opencode-agent @cwl-noema-review @coderabbitai Please review exact current head |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
expected_sha='9d2e31489f476d138062e089509af06cb183cd6d'
echo '== checked-out commit =='
git rev-parse HEAD
git show -s --format='%H%n%P%n%s' HEAD
echo '== PR metadata and checks =='
gh pr view 546 --repo ContextualWisdomLab/fast-mlsirm \
--json number,state,isDraft,headRefOid,baseRefOid,mergeCommit,statusCheckRollup,reviews \
--jq '{number,state,isDraft,headRefOid,baseRefOid,mergeCommit,statusCheckRollup,reviews}'
echo '== changed files at requested head =='
base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD HEAD^)"
git diff --name-status "$base" HEAD
echo '== relevant symbol locations =='
rg -n -C 3 \
'def build_enterprise_issue_facets_calibration_bundle|def fit_scoring_facets_calibration_report|class ScoringFacetsCalibrationReport|ScoringFacetsCalibrationReport' \
. -g '*.py' -g '!**/.git/**' || true
echo '== test and coverage configuration candidates =='
fd -HI '^(pyproject\.toml|pytest\.ini|\.coveragerc|tox\.ini|CHANGELOG\.md)$' . -x sh -c 'echo "--- $1"; sed -n "1,260p" "$1"' sh {} \;Length of output: 42981 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== implementation outline =='
ast-grep outline python/fast_mlsirm/scoring/enterprise_issue/reporting.py --items all --view expanded
ast-grep outline python/fast_mlsirm/scoring/essay/calibration_reporting.py --items all --view expanded
ast-grep outline python/fast_mlsirm/scoring/essay/calibration_report_html.py --items all --view expanded
ast-grep outline tests/test_scoring_enterprise_issue_calibration_reporting.py --items all --view expanded
echo '== enterprise workflow implementation =='
sed -n '1,320p' python/fast_mlsirm/scoring/enterprise_issue/reporting.py
echo '== changed shared trace and HTML replay hunks =='
base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD HEAD^)"
git diff --unified=80 "$base" HEAD -- \
python/fast_mlsirm/scoring/essay/calibration_reporting.py \
python/fast_mlsirm/scoring/essay/calibration_report_html.py \
python/fast_mlsirm/scoring/enterprise_issue/__init__.py
echo '== focused workflow tests =='
sed -n '1,520p' tests/test_scoring_enterprise_issue_calibration_reporting.py
echo '== unresolved review threads =='
gh api graphql \
-F owner='ContextualWisdomLab' \
-F name='fast-mlsirm' \
-F number=546 \
-f query='
query($owner: String!, $name: String!, $number: Int!) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
reviewThreads(first: 100) {
nodes {
isResolved
isOutdated
path
line
comments(first: 10) { nodes { author { login } body } }
}
}
}
}
}' \
--jq '.data.repository.pullRequest.reviewThreads.nodes[] | select((.isResolved | not) and (.isOutdated | not))'
echo '== failed Strix check =='
gh api repos/ContextualWisdomLab/fast-mlsirm/check-runs/92224304941 \
--jq '{name,conclusion,output,details_url:.details_url,started_at,completed_at}'Length of output: 57629 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '== Strix failure annotation =='
gh api repos/ContextualWisdomLab/fast-mlsirm/check-runs/92224304941/annotations \
--paginate \
--jq '.[] | {path,start_line,end_line,annotation_level,message,title,raw_details}'
echo '== Strix failed-job log tail =='
gh run view 30980689991 --repo ContextualWisdomLab/fast-mlsirm --log-failed 2>&1 | tail -n 240 || true
echo '== exact shared fitting delegation =='
sed -n '400,475p' python/fast_mlsirm/scoring/essay/calibration_reporting.py
ast-grep run \
--pattern 'fit_scoring_facets_design($$$)' \
--lang python \
python/fast_mlsirm/scoring/essay/calibration_reporting.py \
python/fast_mlsirm/scoring/calibration_reporting.py
echo '== trace-contract coverage =='
rg -n -C 4 \
'facets_iteration_trace_mismatch|n_iter \+ 1|terminal.*likelihood|terminal.*evaluation|loglik_trace' \
tests python/fast_mlsirm/scoring/essay -g '*.py'
echo '== PR test and coverage-workflow diff =='
base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD HEAD^)"
git diff --unified=30 "$base" HEAD -- tests/test_scoring_enterprise_issue_calibration_reporting.py
fd -HI '(coverage|ci|workflow)' .github . 2>/dev/null | head -n 100
rg -n -C 8 \
'coverage-source-tree|coverage-evidence|fail_under|branch = true|interrogate' \
.github pyproject.toml -g '*.yml' -g '*.yaml' -g '*.toml' 2>/dev/null || true
echo '== changed public documentation and APA doctoring checks =='
sed -n '1,300p' docs/enterprise_issue_facets_calibration_reports.md
sed -n '1,300p' docs/doctoring/enterprise-issue-facets-calibration-reports.mdLength of output: 91221
I found no actionable implementation finding. The workflow calls The workflow returns the existing The trace contract is consistent in report construction and HTML replay. A nonconverged fit can contain No actionable unresolved review threads were returned. The Python, Rust, package, GPU smoke, fuzz, SAST, dependency, OSV, Trivy, CodeQL, Scorecard, and coverage-evidence checks are successful on this head. The documentation, changelog, public docstring, and scientific-limit material are present. I cannot confirm the full required security-gate condition or approve this PR. The exact-head
|
|
@coderabbitai review @opencode-agent @cwl-noema-review Review exact head |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
repo='ContextualWisdomLab/fast-mlsirm'
expected_sha='d111e2b1341b0daab6b813074c7a7dbcf5c24ff4'
echo '== Exact checked-out head =='
git rev-parse HEAD
git show -s --format='commit=%H%nparents=%P%nsubject=%s' HEAD
test "$(git rev-parse HEAD)" = "$expected_sha"
echo '== PR head, check rollup, and reviews =='
gh pr view 546 --repo "$repo" \
--json number,state,isDraft,headRefOid,statusCheckRollup,reviews \
--jq '{number,state,isDraft,headRefOid,statusCheckRollup,reviews}'
echo '== Changed files at exact head =='
base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD HEAD^)"
git diff --name-status "$base" HEAD
echo '== Source and test outlines =='
ast-grep outline python/fast_mlsirm/scoring/enterprise_issue/reporting.py --items all --view expanded
ast-grep outline python/fast_mlsirm/scoring/essay/calibration_reporting.py --items all --view expanded
ast-grep outline python/fast_mlsirm/scoring/essay/calibration_report_html.py --items all --view expanded
ast-grep outline tests/test_scoring_enterprise_issue_calibration_reporting.py --items all --view expanded
ast-grep outline tests/test_scoring_essay_facets_report_html_output_security.py --items all --view expanded
echo '== Enterprise orchestration =='
sed -n '1,340p' python/fast_mlsirm/scoring/enterprise_issue/reporting.py
echo '== Trace validation and HTML output confinement =='
rg -n -C 10 \
'def _validate_iteration_trace_length|def render_essay_facets_calibration_report_html|output_root|resolve\(|is_relative_to|symlink|absolute|relative' \
python/fast_mlsirm/scoring/essay/calibration_reporting.py \
python/fast_mlsirm/scoring/essay/calibration_report_html.py
echo '== Relevant test coverage and Rust integration output root =='
rg -n -C 8 \
'fit_enterprise_issue_facets_calibration_reports|output_root|tmp_path|Rust|n_iter \+ 1|terminal|symlink|absolute|traversal' \
tests/test_scoring_enterprise_issue_calibration_reporting.py \
tests/test_scoring_essay_facets_report_html_output_security.py \
tests/test_scoring_essay_facets_report_html.py
echo '== Unresolved, current review threads =='
gh api graphql \
-F owner='ContextualWisdomLab' -F name='fast-mlsirm' -F number=546 \
-f query='
query($owner: String!, $name: String!, $number: Int!) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
reviewThreads(first: 100) {
nodes {
isResolved isOutdated path line
comments(first: 10) { nodes { author { login } body } }
}
}
}
}
}' \
--jq '.data.repository.pullRequest.reviewThreads.nodes[] | select((.isResolved | not) and (.isOutdated | not))'
echo '== Check runs pinned to exact commit =='
gh api "repos/$repo/commits/$expected_sha/check-runs?per_page=100" \
--jq '.check_runs[] | {name,status,conclusion,details_url,app:.app.slug,started_at,completed_at}'
echo '== Independent approval evidence =='
gh pr view 546 --repo "$repo" --json reviews \
--jq '.reviews[] | {author:.author.login,state,submittedAt,commit:.commit.oid}'Length of output: 92410 🏁 Script executed: #!/bin/bash
set -euo pipefail
repo='ContextualWisdomLab/fast-mlsirm'
sha='d111e2b1341b0daab6b813074c7a7dbcf5c24ff4'
echo '== Shared report helper and Rust-backed fitting boundary =='
ast-grep outline python/fast_mlsirm/scoring/calibration_reporting.py --items all --view expanded
rg -n -C 12 \
'def fit_scoring_facets_calibration_report|fit_scoring_facets_design\(|def fit_scoring_facets_design|FacetsFit' \
python/fast_mlsirm/scoring/calibration_reporting.py \
python/fast_mlsirm/scoring/calibration.py
echo '== Bundle assembly and canonical design ordering =='
ast-grep outline python/fast_mlsirm/scoring/enterprise_issue/calibration.py --items all --view expanded
rg -n -C 10 \
'def build_enterprise_issue_facets_calibration_bundle|designs|sorted\(' \
python/fast_mlsirm/scoring/enterprise_issue/calibration.py
echo '== Focused orchestration test assertions =='
sed -n '100,325p' tests/test_scoring_enterprise_issue_calibration_reporting.py
echo '== Exact-head CI run result =='
gh run view 30989444525 --repo "$repo" \
--json headSha,status,conclusion,jobs \
--jq '{headSha,status,conclusion,jobs:[.jobs[] | {name,status,conclusion,startedAt,completedAt}]}'
echo '== Exact-head Strix run result =='
gh run view 30989443777 --repo "$repo" \
--json headSha,status,conclusion,jobs \
--jq '{headSha,status,conclusion,jobs:[.jobs[] | {name,status,conclusion,startedAt,completedAt}]}'
echo '== Current exact-head check conclusions =='
gh api "repos/$repo/commits/$sha/check-runs?per_page=100" \
--jq '[.check_runs[] | select(.name | IN("python","rust","fuzz","gpu-smoke","package","strix","Semgrep (multi-language SAST)","osv-scan","trivy-fs")) | {name,status,conclusion,app:.app.slug,details_url}]'
echo '== Current reviews, restricted to approvals and exact head =='
gh pr view 546 --repo "$repo" --json reviews \
--jq '[.reviews[] | {author:.author.login,state,commit:.commit.oid,submittedAt}]'Length of output: 17294
I found no actionable implementation issue in the inspected source and focused tests. The enterprise workflow calls The workflow returns only The capped trace contract is consistent. Nonconverged reports accept The HTML renderer resolves the approved root and output path before creating directories. It rejects traversal and absolute escapes outside the root. It resolves existing symlink parents before containment checks. It rejects an existing non-directory root. It rechecks the resolved parent before it writes the report. No unresolved current review thread was returned. I cannot approve this PR yet. The required current-head evidence is incomplete:
Wait for successful exact-head results for those checks. Obtain an independent approval. Then request the final approval review.
|
|
@opencode-agent @cwl-noema-review Final independent exact-head approval requested for |
Pull request was converted to draft
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headd111e2b1341b0daab6b813074c7a7dbcf5c24ff4. -
Head SHA:
d111e2b1341b0daab6b813074c7a7dbcf5c24ff4 -
Workflow run: 30993478444
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 2
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (5 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (5 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (7 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (7 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage Decision
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (5 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (5 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (7 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (7 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test (3 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (3 files)"]
R3 --> V3["targeted test run"]
|
|
@opencode-agent @cwl-noema-review Re-run exact-head coverage and review for |
|
The exact-head OpenCode Central issue ContextualWisdomLab/.github#784 defines the fail-closed repair, and draft PR ContextualWisdomLab/.github#789 adds the bounded classifier and exact-head native peer-check verifier with 81 focused tests and 100% statement/branch coverage. No product test is skipped, no Python substitute is added, and no approval or merge rule is weakened. Keep this PR unmerged until the central workflow integration lands and a fresh exact-head independent approval replaces the current request-changes verdict. |
|
@cwl-noema-review Re-evaluate exact unchanged head |
2 participants
Buyer-visible gap
Enterprise executions can already be assembled into the canonical connected many-facet bundle, and the scoring package exposes domain-neutral names for the canonical Rust-backed report. This PR completes the bounded enterprise orchestration boundary so callers no longer have to manually wire criterion fitting, report IDs, review triggers, and exact bundle provenance.
Implemented boundary
build_enterprise_issue_facets_calibration_bundle()once.fit_scoring_facets_calibration_report()once per canonical criterion design.ScoringFacetsCalibrationReportvalues.Rust trace compatibility
The shared report and HTML replay gates match the Rust estimator contract: a nonconverged fit may retain one terminal post-update likelihood after its
n_iteroptimization iterations. The terminal value remains evidence and is not misreported as another iteration. Other trace cardinalities continue to fail closed.Bounded artifact publication
The standalone facets-report renderer now requires publication within one canonical caller-approved root. It rejects traversal, absolute escapes, existing symlink-parent escapes, and non-directory roots before report writes, then rechecks the canonical parent after directory creation. The current working directory is the default boundary. This is a confinement control, not an operating-system sandbox; the caller must retain authority over the approved directory during publication.
Verification
The focused suite covers deterministic orchestration, execution-order invariance, exact provenance, invalid identifiers, reserved metadata, source privacy, and a realistic connected Rust-backed two-criterion fit. The iteration-cap fixture verifies the terminal likelihood contract and publishes its HTML only beneath an explicit temporary root. Separate adversarial tests cover relative, absolute, symlink, and invalid-root paths. Added and modified production paths are held to focused 100% statement and branch coverage and complete public docstrings. Documentation, APA 7 doctoring, changelog fragments, and rendered
CHANGELOG.mdare included.Exact head
d111e2b1341b0daab6b813074c7a7dbcf5c24ff4passed repository CI30989444525, SAST Semgrep30989444533, Security Scan30989444567, and Strix30989443777. Keep the PR unmerged until current-head automated review, an independent approving review, unresolved-thread policy, and branch protection are satisfied. Do not merge by bypass.Closes #544
Advances #404