feat(Cybersecurity): add DependencyAwarePatchScheduling benchmark - #99
feat(Cybersecurity): add DependencyAwarePatchScheduling benchmark#99oushihsabiy wants to merge 1 commit into
Conversation
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. 质量保证
5. 安全与隐私检查
|
wrh-human
left a comment
wrh-human
left a comment
There was a problem hiding this comment.
Review — DependencyAwarePatchScheduling (PR #99)
Thank you for contributing this benchmark. Cybersecurity patch scheduling is a real engineering problem with clear economic value. After a line-by-line review of all 28 files, the overall design is solid, but several issues were identified that need to be addressed.
1. Domain, Economic Value, and Frontier-Eng Fit ✅
Cybersecurity patch scheduling is a genuine enterprise security operations problem — security teams must balance vulnerability exploitation risk, business downtime costs, and rollback risk to decide which patches to apply and when. The task is anchored by NIST SP 800-40 Rev.4 and CISA BOD 22-01. Scheduling optimization directly maps to minimizing security and business losses, giving the task clear economic value. Starting from a feasible baseline and iteratively improving via a verifier aligns with Frontier-Eng's positioning.
2. Not purely numerical ✅
The agent implements solve(instance), which must decide which patches to select and when to schedule them (start slots), while satisfying dependency closure, maintenance windows, resource capacity, and downtime budget constraints. This is a structured combinatorial scheduling problem, not parameter tuning.
3. Search space ✅
Each patch can be omitted or scheduled at any slot within the horizon. For the large tier (roughly 13-18 patches, 28-37 slots), the search space is approximately (horizon+1)^patches ≈ 10^22, making brute-force search infeasible. The reference exact solver only handles the small tier (5 patches), further demonstrating the search space size.
4. Evaluator and engineering verification ✅
The evaluator independently recomputes all feasibility constraints in validate_solution():
- Dependency closure and prerequisite completion constraints
- Maintenance window containment (assets and services)
- Exclusive change asset overlap detection
- Renewable resource capacity
- Service downtime capacity and cumulative downtime budget
- Patches completing no later than the horizon
The raw objective is independently recomputed in _objective_from_schedule() (security loss + downtime loss + rollback loss), without trusting candidate-reported values. Strict JSON Schema validation is applied to both input and output.
5. Constraint enforcement ✅
Constraints are implemented one by one in the verifier and checked comprehensively. Candidate output must pass schema validation and all constraint checks in validate_solution(). Any failure results in INVALID_SCORE for that case.
6. Baseline experiment
baseline/heuristic.py(deterministic heuristic) andbaseline/weak.py(random) are providedreference/exact.pyprovides an exact solution for the small tier- Deterministic data generation is verified (smoke test checks that two generations match)
- But there is no
baseline/result_log.txtrecording the baseline's actual raw loss, runtime, and other metrics. The baseline's concrete performance and scoring discriminative power cannot be directly verified.
7. Scoring system ✅
_score(baseline, candidate) computes log2(baseline / candidate) for the minimization direction, aggregated via mean. The scoring formula is reasonable: a bounded, interpretable log ratio that effectively measures improvement. The baseline uses dependency-aware benefit-density heuristics, which are non-trivial and improvable; the empty schedule is also feasible (raw loss > 0), providing a lower-bound reference. The exact reference solution for the small tier provides an upper bound on quality.
8. Issues found during full code review
Issue 1 (most critical): Evaluator lacks EVOLVE-BLOCK boundary validation
scripts/init.py has a correct EVOLVE-BLOCK structure — EVOLVE-BLOCK-START (line 8) through EVOLVE-BLOCK-END (line 193) wraps only the solve() function, with imports and the if __name__ == "__main__": block outside the markers. However, the evaluator never checks whether the candidate modified code outside the EVOLVE-BLOCK. A candidate could modify the main block, imports, or other helper logic without detection. Suggestion: validate the candidate's EVOLVE-BLOCK boundary before execution in evaluate(), comparing the outer regions against baseline/heuristic.py or a frozen copy of scripts/init.py.
Issue 2: No test file
There are no test_*.py files. Although a --smoke mode is provided, it is not a pytest suite. At minimum, the following should be added:
- Pass/fail tests for each constraint in
validate_solution() - Boundary tests for the
evaluate_solution()objective computation - A test rejecting EVOLVE-BLOCK boundary violations (after Issue 1 is fixed)
- An end-to-end test of the complete
evaluate()pipeline
Issue 3: Missing baseline run results
Suggestion: create baseline/result_log.txt recording the output of python verification/evaluator.py scripts/init.py --local, including baseline raw loss, candidate raw loss, and scores for each case, plus the reference solver's small-tier results.
Issue 4: Docker isolation configuration is inconsistent with constraint descriptions
benchmark.yaml specifies runtime.isolation: docker, but constraint #9 in constraints.txt explicitly requires "Frontier unified must use its process isolation mode because the benchmark evaluator owns the inner candidate/verifier containers; do not wrap this evaluator in another Docker runtime." If the unified framework wraps the evaluator in Docker according to benchmark.yaml, this conflicts with constraint #9 (double Docker). Suggestion: confirm which configuration field the unified framework actually reads, and eliminate the inconsistency.
Issue 5 (non-blocking): Evaluation seeds hardcoded in the evaluator
SEEDS = [141421, 161803, 173205, 223606, 314159] are directly exposed in verification/evaluator.py. In --local mode (subprocess, no filesystem isolation), a candidate could read the evaluator source and discover the evaluation seeds. The README notes that local mode is only for reviewed code, and publish evaluation uses Docker isolation, so the risk is limited. Suggestion: clarify this in the documentation, or load evaluation seeds from a config file placed inside the Docker image.
Summary
The cybersecurity patch scheduling direction is sound, constraint validation and objective computation are independently reliable, the search space design is reasonable, and Docker and process isolation are in place. Issues 1 (missing EVOLVE-BLOCK boundary validation) and 2 (no test file) are the main items to fix; Issues 3 (missing baseline results) and 4 (Docker config inconsistency) should also be addressed. The review can proceed once the above issues are addressed.
2 participants
Adds an executable dependency-aware cybersecurity patch scheduling benchmark with deterministic instances, independent feasibility verification, risk-based scoring, bilingual documentation, NIST/CISA provenance, calibrated baselines, and Frontier unified metadata.