feat(PrivacyEngineering): add DifferentialPrivacyBudgetAllocation benchmark

[oushihsabiy]

Adds an executable differential privacy budget allocation benchmark with deterministic instances, independent feasibility verification, utility-based scoring, calibrated baseline/reference solvers, bilingual documentation, auditable privacy references, Frontier unified metadata, and PrivacyEngineering domain README files.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new benchmark task titled "Differential Privacy Budget Allocation for Business Analytics." The goal is to optimize the allocation of a finite privacy budget ($\epsilon$) across multiple analytical queries while satisfying constraints related to sensitivity, error bounds, and group fairness.
• Modified File Structure & Modifications:
  • benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/README.md & README_zh-CN.md: Provides the task overview, interface definition, scoring logic, and execution commands.
  • benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/Task.md & Task_zh-CN.md: Detailed task description including JSON schemas for input/output and objective functions.
  • benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/benchmark.yaml: Configuration file defining metadata, seeds, timeouts, and Docker runtime environment.
  • benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/baseline/: Contains heuristic.py and weak.py providing baseline implementations for comparison.
  • benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/frontier_eval/: Integration files for the evaluation framework (constraints, commands, and file lists).
  • benchmarks/PrivacyEngineering/DifferentialPrivacyBudgetAllocation/verification/: (Implied by imports) Contains problem.py and evaluator.py for instance generation and solution validation.

2. AI Content Analysis

• Estimated AI Component: 30%
• Reasoning & Evidence: The overall structure follows a highly standardized template (BenchGen). The baseline/*.py files exhibit boilerplate patterns typical of AI-assisted generation, such as the standard sys.path manipulation and if __name__ == "__main__": blocks. However, the domain-specific logic—specifically the formulation of the DP utility function ($1.0 + \sum business_value \cdot coverage \cdot \log1p(\epsilon) - penalties$) and the fairness ratio constraints—suggests significant human design or highly specialized prompting.

3. Engineering & Economic Assessment

• Engineering Reality Check: This addresses a sophisticated, production-grade problem in Privacy Engineering. Unlike "toy" DP examples that focus on a single mechanism, this task models a portfolio optimization problem. It correctly identifies real-world constraints: minimum $\epsilon$ for utility, maximum $\epsilon$ for privacy, and the $sensitivity/\epsilon$ error relationship. The inclusion of max_group_error_ratio addresses modern algorithmic fairness requirements.
• Economic Value: High. Efficient privacy budget management is a critical bottleneck for data-driven companies (e.g., finance, healthcare) operating under GDPR/CCPA. Optimizing this allocation directly enables higher data utility (revenue) for the same privacy risk (compliance cost).

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: DifferentialPrivacyBudgetAllocation
  • Execution & Dependencies: The README.md clearly documents both local execution (python verification/evaluator.py) and Docker-based execution with a specific image hash, ensuring reproducibility.
• Documentation Quality: High. The documentation provides clear JSON schemas and mathematical definitions of the objective function. The dual-language support is consistent. No significant grammatical errors were detected in the provided diff.
• Organizational Structure: The structure is logical and modular, separating the problem definition (Task.md), the verification logic (verification/), and the baseline implementations.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were found in the diff.
• Absolute Paths: None detected. The scripts use relative path resolution via Path(__file__).resolve().parents[1], which is best practice for portability.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个名为“面向业务分析的差分隐私预算分配”的新基准测试任务。其目标是在满足敏感度、误差边界和组间公平性约束的前提下，优化有限隐私预算 ($\epsilon$) 在多个分析查询中的分配。
• 修改的文件结构与变更摘要:
  • README.md & README_zh-CN.md: 提供任务概览、接口定义、评分逻辑和执行命令。
  • Task.md & Task_zh-CN.md: 详细的任务描述，包括输入/输出的 JSON Schema 和目标函数。
  • benchmark.yaml: 配置文件，定义元数据、种子、超时时间和 Docker 运行环境。
  • baseline/: 包含 heuristic.py 和 weak.py，提供用于对比的基准实现。
  • frontier_eval/: 评测框架的集成文件（约束、命令和文件列表）。
  • verification/: （通过导入体现）包含用于实例生成和方案验证的 problem.py 和 evaluator.py。

2. AI 成分分析

• 预估 AI 含量: 30%
• 判断依据与证据: 整体结构遵循高度标准化的模板 (BenchGen)。baseline/*.py 文件表现出典型的 AI 辅助生成的样板模式，例如标准的 sys.path 处理和 if __name__ == "__main__": 代码块。然而，特定领域的逻辑——特别是差分隐私效用函数 ($1.0 + \sum business_value \cdot coverage \cdot \log1p(\epsilon) - penalties$) 的公式化以及公平性比例约束——表明了显著的人工设计或高度专业化的提示词工程。

3. 工程与经济评估

• 工程现实检验: 该任务解决了隐私工程中一个复杂的生产级问题。与关注单一机制的“玩具级”差分隐私示例不同，此任务建模了一个组合优化问题。它准确识别了现实世界的约束：保证效用的最小 $\epsilon$、保证隐私的最大 $\epsilon$ 以及 $sensitivity/\epsilon$ 的误差关系。引入 max_group_error_ratio 解决了现代算法公平性的要求。
• 经济价值: 高。对于在 GDPR/CCPA 监管下运行的数据驱动型公司（如金融、医疗），高效的隐私预算管理是关键瓶颈。优化此类分配可以直接在相同的隐私风险（合规成本）下实现更高的数据效用（收入）。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: DifferentialPrivacyBudgetAllocation
  • 运行与依赖: README.md 清晰地记录了本地执行命令和基于特定镜像哈希的 Docker 执行命令，确保了可重复性。
• 文档质量: 高。文档提供了清晰的 JSON Schema 和目标函数的数学定义。双语支持保持了一致性。在提供的 diff 中未检测到明显的语法错误。
• 组织结构: 文件组织逻辑清晰且模块化，将问题定义 (Task.md)、验证逻辑 (verification/) 和基准实现进行了有效分离。

5. 安全与隐私检查

• 敏感文件: 未发现异常。Diff 中未发现 .env、API 密钥或 IDE 配置文件。
• 绝对路径: 未检测到。脚本使用 Path(__file__).resolve().parents[1] 进行相对路径解析，符合可移植性最佳实践。

[wrh-human]

Review — DifferentialPrivacyBudgetAllocation (PR #100)

Thank you for contributing this benchmark. Differential privacy budget allocation is a genuine privacy engineering problem with clear economic value. After a complete line-by-line review of all files, the overall design is solid, but several issues were identified that need to be addressed.

---

1. Domain, Economic Value, and Frontier-Eng Fit ✅

Differential privacy budget allocation is a real data analytics engineering problem — privacy budget (epsilon) is a scarce resource that must be distributed across multiple analytics queries while satisfying per-query accuracy requirements, a total budget cap, and group fairness constraints. The task is anchored by NIST SP 800-226 and Dwork & Roth's differential privacy work. A good budget allocation directly maps to maximizing data utility, giving the task clear economic value. Starting from a feasible baseline and iteratively improving via a verifier aligns with Frontier-Eng's positioning.

2. Not purely numerical ✅

The agent implements solve(instance), which must allocate an epsilon value to each query while satisfying per-query epsilon bounds, maximum error, total budget, and group fairness constraints. This is a structured constrained allocation problem involving joint tradeoffs across business value, population coverage, sensitivity, and fairness — not parameter tuning.

3. Search space ✅

Each query's epsilon is a continuous value in [epsilon_min, epsilon_max], subject to total budget and group fairness constraints. The continuous allocation space across roughly 14-18 queries cannot be brute-forced. The objective function is concave (the log1p term is concave, and the error penalty term -sensitivity/eps is also concave), which theoretically enables convex optimization and provides a meaningful improvement path for agents.

4. Evaluator and engineering verification ✅

The evaluator independently recomputes all feasibility constraints in validate_solution():

• Allocations must cover exactly the query IDs
• epsilon within [epsilon_min, epsilon_max]
• error = sensitivity / epsilon does not exceed max_error
• Total epsilon does not exceed epsilon_total
• Group average error fairness ratio does not exceed max_group_error_ratio
• Finite value checks (not bool, finite, not NaN)

The raw objective is independently recomputed in _objective() without trusting candidate-reported values. Strict JSON Schema validation is applied to both input and output. The candidate runs in a subprocess (process_runner.py with timeout, output limits, and process group cleanup), and publish evaluation uses Docker isolation (pinned image SHA, non-root, network disabled, read-only filesystem, memory/CPU/process limits).

5. Constraint enforcement ✅

Constraints are implemented one by one in the verifier and checked comprehensively. Candidate output must pass schema validation and all constraint checks in validate_solution(). Any failure results in INVALID_SCORE (-1e18) for that case.

6. Baseline experiment ⚠️ Partially missing

• baseline/heuristic.py (deterministic heuristic) and baseline/weak.py (random) are provided
• reference/exact.py provides a reference solver
• Deterministic data generation is verified (smoke test checks that two generations match)
• But there is no baseline/result_log.txt recording the baseline's actual raw utility and runtime. The baseline's concrete performance and scoring discriminative power cannot be directly verified.

7. Scoring system ✅

_score(baseline, candidate) computes log2(candidate / baseline) for the maximization direction, aggregated via mean. The scoring formula is reasonable: a bounded, interpretable log ratio that effectively measures improvement. The baseline uses fairness repair + greedy allocation, which is non-trivial and improvable. The concave objective provides a clear improvement path for better convex optimization solvers, so the scoring discriminative power is well supported.

8. Issues to address

Issue 1 (most critical): Evaluator lacks EVOLVE-BLOCK boundary validation

scripts/init.py has a correct EVOLVE-BLOCK structure — EVOLVE-BLOCK-START (line 8) through EVOLVE-BLOCK-END (line 76) wraps only the solve() function, with imports and the if __name__ == "__main__": block outside the markers. However, the evaluator contains no EVOLVE-related validation logic and never checks whether the candidate modified code outside the EVOLVE-BLOCK. A candidate could modify the main block, imports, or other helper logic without detection. Suggestion: validate the candidate's EVOLVE-BLOCK boundary before execution in evaluate().

Issue 2: No test file

There are no test_*.py files. Although a --smoke mode is provided, it is not a pytest suite. At minimum, the following should be added:

• Pass/fail tests for each constraint in validate_solution()
• Boundary tests for the evaluate_solution() objective computation
• A test rejecting EVOLVE-BLOCK boundary violations (after Issue 1 is fixed)
• An end-to-end test of the complete evaluate() pipeline

Issue 3: Missing baseline run results

Suggestion: create baseline/result_log.txt recording the output of python verification/evaluator.py scripts/init.py --local, including baseline raw utility, candidate raw utility, and scores for each case, plus the reference solver's results.

Issue 4: Docker isolation configuration is inconsistent with constraint descriptions

benchmark.yaml specifies runtime.isolation: docker, but constraint #9 in constraints.txt explicitly requires "Frontier unified must use its process isolation mode because the benchmark evaluator owns the inner candidate/verifier containers; do not wrap this evaluator in another Docker runtime." If the unified framework wraps the evaluator in Docker according to benchmark.yaml, this conflicts with constraint #9 (double Docker). Suggestion: confirm which configuration field the unified framework actually reads, and eliminate the inconsistency.

Issue 5 (non-blocking): Evaluation seeds hardcoded in the evaluator

SEEDS = [210001, 210013, ...] at line 22 of verification/evaluator.py are directly exposed in the source. In --local mode (subprocess, no filesystem isolation), a candidate could read the evaluator source and discover the evaluation seeds. The README notes that local mode is only for reviewed code, and publish evaluation uses Docker isolation, so the risk is limited. Suggestion: clarify this in the documentation, or load evaluation seeds from a config file placed inside the Docker image.

---

Summary

The differential privacy budget allocation direction is sound, constraint validation and objective computation are independently reliable, the concave objective provides a clear optimization path, and Docker and process isolation are in place. Issues 1 (missing EVOLVE-BLOCK boundary validation), 2 (no test file), and 3 (missing baseline results) are the main items to fix; Issue 4 (Docker config inconsistency) should also be addressed. The review can proceed once the above issues are addressed.