Add FJSP-WF task for manufacturing scheduling

[y-ji24]

Summary

This PR adds a new Frontier-Eng task for Flexible Job Shop Scheduling with Worker Flexibility (FJSP-WF) under the Manufacturing domain.

Highlights

• Adds a manufacturing scheduling task based on official FJSSP-W benchmark instances.
• Uses solver/scheduler.py as the editable Agent artifact with EVOLVE-BLOCK protection.
• Provides a deterministic evaluator with feasibility validation and relative scoring.
• Includes a baseline scheduler for reproducible evaluation.
• Adds task documentation, references, benchmark metadata, and Frontier-Eng integration files.

Verification

• Baseline and solver schedules are feasible on all benchmark instances.
• Evaluator produces deterministic scores.
• Benchmark metadata and instance files have been validated.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a significant update to the evaluation framework, including a new scoring metric ("Medal Score"), a lightweight benchmark subset (v1-lite), and a new complex optimization task (Manufacturing/FJSP-WF).
• Modified File Structure & Modifications:
  • .gitignore: Whitelisted leaderboard/*.csv and the new v1_lite.yaml configuration.
  • README.md & README_zh-CN.md: Updated with the 2026-06-30 news, detailed explanation of the Medal Score metric, introduction of the v1-lite subset, and a revamped leaderboard table.
  • benchmarks/Manufacturing/FJSP-WF/README.md: Provided task overview, engineering background, and quick-start commands.
  • benchmarks/Manufacturing/FJSP-WF/Task.md: Detailed technical specification of the Flexible Job Shop Scheduling problem, including constraints, I/O formats, and validation rules.
  • benchmarks/Manufacturing/FJSP-WF/baseline/scheduler.py: Implemented a greedy Earliest Start Time + Shortest Processing Time (EST+SPT) baseline algorithm.
  • benchmarks/Manufacturing/FJSP-WF/data/benchmark_instances.json: Registry for 30 official and 3 synthetic instances.

2. AI Content Analysis

• Estimated AI Component: 25%
• Reasoning & Evidence: The documentation (Task.md) and the baseline code (scheduler.py) exhibit high structural consistency and professional phrasing typical of AI-assisted technical writing. The solve_instance function follows a very standard, clean boilerplate pattern for greedy algorithms. However, the domain-specific nuances of the FJSP-WF problem (worker-machine eligibility matrices) and the specific integration with the frontier_eval framework suggest significant human oversight and domain expertise.

3. Engineering & Economic Assessment

• Engineering Reality Check: High. FJSP-WF is a classic NP-hard problem in operations research. The PR moves beyond "toy" examples by incorporating 30 official instances from the GECCO 2026 competition. The validation logic is robust, checking for job precedence, machine/worker overlaps, and eligibility constraints, which are critical for production-grade scheduling.
• Economic Value: High. Production scheduling optimization is a multi-billion dollar challenge in manufacturing. Improving makespan by even small percentages directly impacts throughput and energy efficiency. The introduction of the "Medal Score" also provides higher economic signal by rewarding models that reach the "frontier" of known optimal solutions rather than rewarding noise in sub-optimal ranges.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Manufacturing/FJSP-WF
  • Execution & Dependencies: The README.md and Task.md provide explicit CLI commands for both standalone evaluation and framework-level execution. Environment dependencies are limited to the Python standard library, ensuring high portability.
• Documentation Quality: Excellent. The documentation covers the "Why" (Engineering Background) as well as the "How".
  • Correction: In Task.md, the header ## Instance Data is duplicated consecutively.
• Organizational Structure: Logical and scalable. The task follows a clear hierarchy: baseline/, data/, solver/, and verification/, which is consistent with the existing repository architecture.

5. Security & Privacy Check

• Sensitive Files: Clean. The .gitignore was proactively updated to prevent accidental leaks of local logs or IDE configs.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and environment variables (FJSPWF_BENCHMARK_JSON) to handle paths dynamically.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 为评测框架引入了重大更新，包括新的评分维度（“勋章分” Medal Score）、轻量化评测子集（v1-lite）以及一个新的复杂优化任务（Manufacturing/FJSP-WF）。
• 修改的文件结构与变更摘要:
  • .gitignore: 将 leaderboard/*.csv 和新的 v1_lite.yaml 配置加入白名单。
  • README.md & README_zh-CN.md: 更新了 2026-06-30 的新闻动态，详细解释了 Medal Score 指标，介绍了 v1-lite 子集，并更新了排行榜表格。
  • benchmarks/Manufacturing/FJSP-WF/README.md: 提供了任务概览、工程背景和快速启动命令。
  • benchmarks/Manufacturing/FJSP-WF/Task.md: 详细说明了柔性车间调度（FJSP）问题的技术规范，包括约束条件、输入输出格式和验证规则。
  • benchmarks/Manufacturing/FJSP-WF/baseline/scheduler.py: 实现了一个基于最早开始时间+最短处理时间（EST+SPT）的贪心基准算法。
  • benchmarks/Manufacturing/FJSP-WF/data/benchmark_instances.json: 注册了 30 个官方实例和 3 个合成实例。

2. AI 成分分析

• 预估 AI 含量: 25%
• 判断依据与证据: 文档（Task.md）和基准代码（scheduler.py）表现出 AI 辅助技术写作典型的高结构一致性和专业措辞。solve_instance 函数遵循了非常标准、整洁的贪心算法模板。然而，FJSP-WF 问题的领域特定细节（如工人-机器资格矩阵）以及与 frontier_eval 框架的特定集成表明存在显著的人工监督和领域专业知识。

3. 工程与经济评估

• 工程现实检验: 高。FJSP-WF 是运筹学中经典的 NP-hard 问题。该 PR 通过引入 GECCO 2026 竞赛的 30 个官方实例，超越了“玩具级”示例。验证逻辑稳健，检查了作业优先级、机器/工人重叠以及资格约束，这些对于生产级调度至关重要。
• 经济价值: 高。生产调度优化是制造业中价值数十亿美元的挑战。即使是微小的完工时间（makespan）改进也能直接影响吞吐量和能源效率。引入“勋章分”通过奖励达到已知最优解“前沿”的模型，而非奖励次优区间的噪声，提供了更高的经济信号。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: Manufacturing/FJSP-WF
  • 运行与依赖: README.md 和 Task.md 提供了明确的 CLI 命令，支持独立评估和框架级执行。环境依赖仅限于 Python 标准库，确保了高移植性。
• 文档质量: 优秀。文档涵盖了“为什么”（工程背景）以及“怎么做”。
  • 纠错: 在 Task.md 中，## Instance Data 标题连续重复出现了两次。
• 组织结构: 逻辑清晰且具备可扩展性。任务遵循明确的层级结构：baseline/、data/、solver/ 和 verification/，与现有仓库架构保持一致。

5. 安全与隐私检查

• 敏感文件: 未发现异常。.gitignore 已主动更新，以防止本地日志或 IDE 配置意外泄露。
• 绝对路径: 未检测到。代码使用 Path(__file__).resolve() 和环境变量（FJSPWF_BENCHMARK_JSON）来动态处理路径。

[wrh-human]

Review — FJSP-WF (PR #89)

Thank you for contributing this benchmark! Flexible Job Shop Scheduling with Worker Flexibility is a real manufacturing problem with clear economic value. The documentation is thorough and the data sources are reliable. After a line-by-line review of all code, the following issues were identified.

---

Evaluation by dimension

1. Domain, Economic Value, and Frontier-Eng Fit ✅

FJSP-WF is a genuine production scheduling problem — factories must decide job sequencing, machine assignment, and worker allocation every day. The task uses 30 official instances from the GECCO FJSSP-WU competition with traceable data sources. Scheduling optimization directly translates to improved equipment utilization and shorter delivery times, giving the task clear economic value.

2. Not purely numerical ✅

The agent modifies the scheduling algorithm logic in solver/scheduler.py (machine assignment, operation sequencing, worker allocation strategies), not numerical parameters. This requires understanding the problem structure of job shop scheduling.

3. Search space ✅

FJSP-WF is NP-hard. The baseline uses a greedy strategy with clear room for improvement.

4. Evaluator and engineering verification ✅

validate_schedule() verifies machine conflicts, worker conflicts, job precedence, eligibility constraints, and time consistency across 14 checks — comprehensive coverage. parse_fjs_file() correctly parses GECCO .fjs format (verified against a sample instance file). Both .fjs and .fjswf formats are supported.

5. Constraint enforcement ✅

Constraints are enforced incrementally in validate_schedule(). Violations mark the schedule as valid=False and return score=None.

6. Baseline experiment ⚠️ Suggested improvement

baseline/scheduler.py is included and differs from solver/scheduler.py only in comments and EVOLVE markers; the core algorithm logic is identical.

7. Scoring system

baseline_makespan / agent_makespan, simple average (statistics.fmean), intuitive and understandable.

---

Issues to address

Issue 1 (most critical): Candidate runs in-process — no process isolation

The evaluator uses importlib.util.spec_from_file_location() (line 172) to import and execute the candidate in the same process as the evaluator (line 712). The candidate can read/modify evaluator state, import any module, and access the network. No subprocess isolation, no resource limits. A candidate crash can take down the evaluator. All other benchmarks use subprocess isolation.

Suggestion: execute the candidate in a subprocess (e.g., subprocess.Popen), communicate via JSON, and set RLIMIT_NPROC, RLIMIT_AS, RLIMIT_CPU with start_new_session=True.

Issue 2 (most critical): Missing EVOLVE-BLOCK boundary validation

solver/scheduler.py contains EVOLVE-BLOCK-START/END markers at lines 83-215. grep -c "EVOLVE" on the evaluator returns 0 — it never checks whether the candidate modified code outside the markers.

Suggestion: add EVOLVE-BLOCK boundary validation before importing the candidate source.

Issue 3: TASK_DETAILS.md not updated

PR #89 does not add the Manufacturing domain or FJSP-WF task to TASK_DETAILS.md. Users cannot discover this benchmark from the main task index.

Suggestion: add the Manufacturing domain and FJSP-WF task entry to TASK_DETAILS.md.

Issue 4: No test file

No test files exist. At minimum, the following should be added:

• A test verifying the baseline produces valid schedules on all instances
• A test rejecting EVOLVE-BLOCK boundary violations
• Tests for validate_schedule() correctly detecting various constraint violations

---

Non-blocking suggestions

1. Add baseline run results: Create baseline/result_log.json recording the output of python verification/evaluator.py solver/scheduler.py across all instances, so reviewers can directly verify baseline performance.

2. Guard gap in validate_schedule() processing time check: The if expected_duration > 0 guard at line 500 skips the check when expected_duration == 0 (worker not available for this combination), allowing the candidate to assign any non-zero duration without detection. This is a very low-probability edge case but represents a theoretical consistency gap. Suggestion: also report an error when processing time is 0 but the assigned duration is non-zero.

---

FJSP-WF has the right direction, thorough documentation, and reliable data. The review can proceed once the above issues are addressed.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark task, FJSP-WF (Flexible Job Shop Scheduling with Worker Flexibility), to the frontier_eval framework. It models a real-world manufacturing optimization problem where operations must be assigned to both machines and workers with varying skill levels to minimize makespan.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md: Added FJSP-WF to the Manufacturing category table.
  • benchmarks/Manufacturing/FJSP-WF/README.md: Provided a high-level overview, benchmark structure, and quick-start execution commands.
  • benchmarks/Manufacturing/FJSP-WF/Task.md: Detailed technical specification, including problem formulation, I/O formats, validation rules, and scoring logic.
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: Recorded baseline performance metrics across 30 official instances.
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: Implemented a greedy baseline algorithm using Earliest Start Time + Shortest Processing Time (EST+SPT).

2. AI Content Analysis

• Estimated AI Component: 25%
• Reasoning & Evidence:
  • The documentation in Task.md is highly structured and follows a template typical of AI-assisted technical writing (e.g., the "Audience and Assumptions" and "Why Worker Flexibility Matters" sections).
  • The solution.py script uses very standard Python boilerplate for natural sorting (_natural_key) and path resolution, which are common AI-generated patterns.
  • However, the domain-specific logic for FJSP-WF constraints (machine vs. worker eligibility) and the integration with the GECCO 2026 competition dataset indicate significant human engineering and domain expertise.

3. Engineering & Economic Assessment

• Engineering Reality Check: High. This is not a toy problem. It utilizes the official GECCO 2026 FJSSP-WU Competition dataset. It addresses multi-dimensional constraints (precedence, machine eligibility, and worker skill levels) that are representative of production-grade industrial engineering.
• Economic Value: High. Production scheduling is a multi-billion dollar challenge in manufacturing. Improving makespan by even small percentages directly correlates to increased throughput and reduced operational costs. Providing a standardized benchmark for AI agents to solve these NP-hard problems has significant research and commercial value.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: FJSP-WF (referenced as Manufacturing/FJSP-WF in the framework).
  • Execution & Dependencies: The README.md clearly documents the execution commands for both the evaluator and the frontier_eval framework. It specifies a "Pure Python" requirement, minimizing environment dependency issues.
• Documentation Quality: Good, but with minor errors.
  • Error: In Task.md, there is a redundant/malformed header: ## Input / Output## Input / Output Specification.
  • The documentation is otherwise excellent, providing clear context for developers who may not be experts in operations research.
• Organizational Structure: Logical and Scalable. The separation of baseline, data, scripts, and verification follows best practices for benchmark repositories.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were detected.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and environment variables to handle file paths dynamically.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 为 frontier_eval 框架引入了一个新的工程基准测试任务 —— FJSP-WF（具备工人灵活性的柔性作业车间调度）。该任务模拟了现实世界中的制造优化问题，要求将工序分配给具有不同技能水平的机器和工人，以最小化完工时间（Makespan）。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md: 在 Manufacturing（制造）类别表格中添加了 FJSP-WF。
  • benchmarks/Manufacturing/FJSP-WF/README.md: 提供了高层概述、基准结构和快速启动执行命令。
  • benchmarks/Manufacturing/FJSP-WF/Task.md: 详细的工程技术规范，包括问题公式化、输入/输出格式、验证规则和评分逻辑。
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: 记录了 30 个官方实例的基准性能指标。
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: 实现了一个基于最早开始时间 + 最短加工时间 (EST+SPT) 的贪心基准算法。

2. AI 成分分析

• 预估 AI 含量: 25%
• 判断依据与证据:
  • Task.md 中的文档结构非常严整，符合 AI 辅助技术写作的典型模板（例如“受众与假设”和“为什么工人灵活性很重要”章节）。
  • solution.py 脚本使用了非常标准的 Python 模板代码进行自然排序 (_natural_key) 和路径解析，这是常见的 AI 生成模式。
  • 然而，针对 FJSP-WF 约束（机器与工人资格）的特定领域逻辑以及与 GECCO 2026 竞赛数据集的集成，表明了显著的人工工程投入和领域专业知识。

3. 工程与经济评估

• 工程现实检验: 高。这绝非玩具示例。它使用了官方的 GECCO 2026 FJSSP-WU 竞赛数据集，处理了代表生产级工业工程的多维约束（前序关系、机器资格和工人技能水平）。
• 经济价值: 高。生产调度是制造业中价值数十亿美元的挑战。即使完工时间缩短很小的比例，也能直接转化为吞吐量的增加和运营成本的降低。为 AI Agent 提供一个标准化的基准来解决这些 NP-hard 问题具有显著的研究和商业价值。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: FJSP-WF（在框架中引用为 Manufacturing/FJSP-WF）。
  • 运行与依赖: README.md 清晰地记录了评估器和 frontier_eval 框架的运行命令。它明确了“纯 Python”要求，最大限度地减少了环境依赖问题。
• 文档质量: 良好，但存在细微错误。
  • 错误: 在 Task.md 中，存在一个冗余/格式错误的标题：## Input / Output## Input / Output Specification。
  • 除此之外，文档非常出色，为非运筹学专家的开发人员提供了清晰的背景信息。
• 组织结构: 逻辑清晰且具备可扩展性。baseline、data、scripts 和 verification 的分离符合基准测试仓库的最佳实践。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未检测到 .env、API 密钥或 IDE 特定配置。
• 绝对路径: 未检测到。代码使用 Path(__file__).resolve() 和环境变量来动态处理文件路径。

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🤖 LLM 调用失败: HTTPSConnectionPool(host='litellm-dev.vida.app', port=443): Read timed out. (read timeout=120)

[wrh-human]

Review — FJSP-WF (PR #89) — Updated Re-review

Thank you for the timely updates! Of the five previously raised issues, process isolation, test files, baseline run results, and TASK_DETAILS.md have all been correctly fixed. However, a critical implementation issue was identified during re-review.

---

Correctly fixed items

Process isolation (in-process → subprocess): ✅ Fixed. Added _candidate_runner.py and _run_candidate_subprocess(). The candidate now runs in a separate subprocess with JSON communication, no longer sharing a process with the evaluator.

Test files: ✅ Fixed. Added test_evolve_block.py (4 tests), test_baseline.py (3 tests), and test_evaluator.py (4 tests), totaling 11 tests.

Baseline run results: ✅ Fixed. baseline/result_log.json contains per-instance details for all 30 instances.

TASK_DETAILS.md: ✅ Fixed. The Manufacturing domain and FJSP-WF task entry have been added.

---

Issue to address

The EVOLVE-BLOCK validation compares the candidate against init.py itself, making the check completely ineffective in the normal evaluation flow

Lines 793-794:

original_solver_path = benchmark_dir / "scripts" / "init.py"
evolve_ok, evolve_msg = _check_evolve_block(candidate_path, original_solver_path)

When running normally (python verification/evaluator.py scripts/init.py):

• candidate_path resolves to scripts/init.py
• original_solver_path also resolves to scripts/init.py
• _check_evolve_block reads the same file twice and compares the two reads

The content of the same file is always equal, so the check always passes. Even if the agent modifies code outside the EVOLVE-BLOCK in init.py (which should be prohibited), _check_evolve_block reads the same modified file as both candidate and original — they are still equal, and the check still passes.

The test test_baseline_fails_evolve_check passes because it passes baseline/solution.py — a file that differs from init.py. But in the real workflow where scripts/init.py is passed, the validation is entirely ineffective. The root cause is that the reference on line 794 is not the frozen baseline, but init.py itself — the very file being modified.

Suggestion: replace original_solver_path on line 794 with the previously resolved baseline_path (line 788), so the validation compares the candidate against the frozen baseline/solution.py. This way, even if the agent modifies init.py's outer code, the validation will detect the difference.

---

Additional suggestions

1. subprocess.run lacks resource limits: No preexec_fn is set. Consider adding RLIMIT_NPROC, RLIMIT_AS, and RLIMIT_CPU as defense in depth.
2. The parameter name original_path in _check_evolve_block is misleading: The actual value passed is scripts/init.py. Consider renaming it to reference_path to avoid confusion with the semantics of "original copy."

---

The review can proceed once the above issues are addressed!

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark task: FJSP-WF (Flexible Job Shop Scheduling with Worker Flexibility). It is designed to evaluate an AI agent's ability to optimize complex manufacturing schedules using real-world competition data (GECCO 2026).
• Modified File Structure & Modifications:
  • TASK_DETAILS.md: Registered the new FJSP-WF task under the Manufacturing category.
  • benchmarks/Manufacturing/FJSP-WF/README.md: Provided high-level overview, quick start commands, and framework integration instructions.
  • benchmarks/Manufacturing/FJSP-WF/Task.md: Detailed technical specification, including problem formulation, I/O specs, and scoring logic.
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: Stored baseline performance metrics for 30 official instances.
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: Implemented a greedy EST+SPT (Earliest Start Time + Shortest Processing Time) baseline algorithm.
  • benchmarks/Manufacturing/FJSP-WF/scripts/init.py (referenced): The entry point for AI agents to modify and evolve the scheduling logic.
  • benchmarks/Manufacturing/FJSP-WF/verification/evaluator.py (referenced): The core validation engine for checking schedule feasibility and calculating scores.

2. AI Content Analysis

• Estimated AI Component: 25%
• Reasoning & Evidence: The documentation (Task.md) is exceptionally detailed and follows a specific pedagogical structure likely guided by a template. The baseline code in solution.py uses very standard Python patterns for greedy algorithms. While the domain-specific logic (handling the 3D assignment of Job-Machine-Worker) is precise, the boilerplate for argument parsing and file I/O exhibits AI-typical clean, standard formatting. The "Engineering Background" section in Task.md feels human-refined to ensure it meets the specific "Engineering Reality" criteria of this repository.

3. Engineering & Economic Assessment

• Engineering Reality Check: High. This is a production-grade problem. Unlike basic Job Shop Scheduling, this includes "Worker Flexibility," which accounts for varying skill levels and cross-training—a critical real-world constraint. It uses the official GECCO 2026 competition dataset, ensuring the problem complexity is representative of current industrial research.
• Economic Value: High. In manufacturing, a 1% reduction in makespan (total completion time) directly correlates to increased throughput and reduced operational costs. Providing a framework for AI to optimize these schedules has significant potential for technical debt reduction in legacy ERP/MES systems.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Manufacturing/FJSP-WF
  • Execution & Dependencies: The README.md clearly documents the execution commands for both the standalone evaluator and the frontier_eval framework. It explicitly states that only the Python standard library is allowed, minimizing environment installation friction.
• Documentation Quality: Good, but with a minor formatting error. In Task.md, there is a redundant header: ## Input / Output## Input / Output Specification. Otherwise, the documentation is comprehensive, providing clear constraints and scoring formulas.
• Organizational Structure: Excellent. The separation of baseline/, data/, scripts/, and verification/ follows a logical, modular structure that is easily scalable for additional manufacturing benchmarks.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or __pycache__ directories were detected in the diff.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and relative pathing to locate data files, which is correct for portability.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个新的工程基准测试任务：FJSP-WF（考虑工人灵活性的柔性作业车间调度）。旨在利用真实的竞赛数据（GECCO 2026）评估 AI Agent 优化复杂制造调度方案的能力。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md: 在 Manufacturing 类别下注册了新的 FJSP-WF 任务。
  • benchmarks/Manufacturing/FJSP-WF/README.md: 提供了高层概述、快速启动命令和框架集成说明。
  • benchmarks/Manufacturing/FJSP-WF/Task.md: 详细的技术规范，包括问题公式化、输入/输出规范和评分逻辑。
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: 存储了 30 个官方实例的基准性能指标。
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: 实现了一个贪心的 EST+SPT（最早开始时间 + 最短处理时间）基准算法。
  • benchmarks/Manufacturing/FJSP-WF/scripts/init.py (引用): AI Agent 修改和演化调度逻辑的入口文件。
  • benchmarks/Manufacturing/FJSP-WF/verification/evaluator.py (引用): 用于检查调度可行性并计算得分的核心验证引擎。

2. AI 成分分析

• 预估 AI 含量: 25%
• 判断依据与证据: 文档 (Task.md) 非常详尽，且遵循了可能由模板引导的特定教学结构。solution.py 中的基准代码使用了非常标准的贪心算法 Python 模式。虽然处理“工件-机器-工人”三维分配的领域逻辑很精确，但参数解析和文件 I/O 的样板代码表现出 AI 典型的整洁、标准格式。Task.md 中的“工程背景”部分经过了人工精修，以确保符合本仓库的“工程现实”标准。

3. 工程与经济评估

• 工程现实检验: 高。这是一个生产级别的工程问题。与基础的作业车间调度不同，它包含了“工人灵活性”，考虑了不同的技能水平和交叉培训——这是一个关键的现实约束。它使用了 GECCO 2026 竞赛的官方数据集，确保了问题的复杂性代表了当前的工业研究水平。
• 经济价值: 高。在制造业中，完工时间（makespan）减少 1% 直接对应于产量的增加和运营成本的降低。为 AI 优化这些调度提供框架，对于减少传统 ERP/MES 系统中的技术债务具有重大潜力。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: Manufacturing/FJSP-WF
  • 运行与依赖: README.md 清晰地记录了独立评估器和 frontier_eval 框架的运行命令。它明确规定仅允许使用 Python 标准库，最大限度地减少了环境安装的摩擦。
• 文档质量: 良好，但存在一处微小的格式错误。在 Task.md 中，有一个冗余标题：## Input / Output## Input / Output Specification。除此之外，文档非常全面，提供了清晰的约束和评分公式。
• 组织结构: 优秀。baseline/、data/、scripts/ 和 verification/ 的划分遵循逻辑化、模块化的结构，易于扩展其他制造类基准测试。

5. 安全与隐私检查

• 敏感文件: 未发现异常。Diff 中未检测到 .env、API 密钥或 __pycache__ 目录。
• 绝对路径: 未检测到。代码使用 Path(__file__).resolve() 和相对路径来定位数据文件，符合可移植性要求。

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark task: FJSP-WF (Flexible Job Shop Scheduling with Worker Flexibility). It is designed to evaluate AI agents' ability to optimize complex manufacturing schedules by assigning operations to machines and workers while minimizing makespan.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md: Added FJSP-WF to the Manufacturing category table.
  • benchmarks/Manufacturing/FJSP-WF/README.md: Provided a high-level overview, benchmark structure, and quick-start commands.
  • benchmarks/Manufacturing/FJSP-WF/Task.md: Detailed the engineering background, problem formulation (NP-hard optimization), I/O specifications, and validation rules.
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: Included baseline performance metrics for 30 official instances.
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: Implemented a greedy EST+SPT (Earliest Start Time + Shortest Processing Time) baseline algorithm.

2. AI Content Analysis

• Estimated AI Component: 35%
• Reasoning & Evidence:
  • Documentation Structure: The Task.md and README.md follow a highly standardized, professional template common in AI-generated technical documentation (e.g., "Audience and Assumptions", "What This Benchmark Evaluates").
  • Formatting Errors: There are clear signs of AI-assisted "hallucination" or copy-paste errors in the Markdown headers, such as the repeated ## Instance Data and the mangled ## Input / Output## Input / Output Specification.
  • Code Style: The solution.py uses very standard, clean boilerplate for argument parsing and file I/O, though the core scheduling logic shows domain-specific nuance related to the FJSP-WF constraints.

3. Engineering & Economic Assessment

• Engineering Reality Check: Production-Grade. Unlike simple "Job Shop" puzzles, this benchmark incorporates "Worker Flexibility," a critical real-world constraint where worker skill levels vary. It utilizes the official GECCO 2026 competition dataset, ensuring the problem complexity is representative of industrial operations research.
• Economic Value: High. In manufacturing, a 1-5% reduction in makespan (total completion time) directly correlates to increased throughput and reduced operational costs. This benchmark provides a rigorous environment to test if AI agents can outperform traditional heuristics in combinatorial optimization.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Manufacturing/FJSP-WF
  • Execution & Dependencies: The README clearly documents execution commands for both standalone evaluation and the frontier_eval framework. It specifies a "Pure Python" requirement, minimizing environment friction.
• Documentation Quality: Good but needs polish. The technical explanation of the FJSP-WF problem is excellent. However, there are redundant headers in Task.md (e.g., ## Instance Data appears twice) and a formatting error in the Input / Output Specification header.
• Organizational Structure: Logical and Scalable. The separation of baseline/, data/, and verification/ follows best practices for benchmark repositories.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific files (.vscode/) were detected. The result_log.json is appropriate for a benchmark baseline.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and relative pathing to locate data files, which is robust for cross-system execution.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个新的工程基准测试任务：FJSP-WF（考虑工人灵活性的柔性作业车间调度）。旨在评估 AI Agent 在最小化完工时间（makespan）的目标下，通过优化机器和工人的分配来处理复杂制造调度问题的能力。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md: 在 Manufacturing 类别表格中添加了 FJSP-WF。
  • benchmarks/Manufacturing/FJSP-WF/README.md: 提供了高层概述、基准结构和快速启动命令。
  • benchmarks/Manufacturing/FJSP-WF/Task.md: 详细说明了工程背景、问题公式化（NP-hard 优化）、输入/输出规范和验证规则。
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: 包含了 30 个官方实例的基准性能指标。
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: 实现了一个贪心的 EST+SPT（最早开始时间 + 最短加工时间）基准算法。

2. AI 成分分析

• 预估 AI 含量: 35%
• 判断依据与证据:
  • 文档结构: Task.md 和 README.md 遵循了高度标准化的专业模板，这是 AI 生成技术文档的典型特征（如“受众与假设”、“本基准评估的内容”等章节）。
  • 格式错误: Markdown 标题中存在明显的 AI 辅助编辑或复制粘贴错误，例如重复的 ## Instance Data 以及损坏的标题 ## Input / Output## Input / Output Specification。
  • 代码风格: solution.py 在参数解析和文件 I/O 方面使用了非常标准的模板化代码，但核心调度逻辑体现了与 FJSP-WF 约束相关的特定领域知识。

3. 工程与经济评估

• 工程现实检验: 生产级。与简单的“车间作业”谜题不同，该基准引入了“工人灵活性”，这是现实世界中工人技能水平存在差异的关键约束。它采用了 GECCO 2026 竞赛的官方数据集，确保了问题复杂度足以代表工业运筹学。
• 经济价值: 高。在制造业中，完工时间（总完工时间）减少 1-5% 直接对应于吞吐量的提高和运营成本的降低。该基准提供了一个严格的环境，用于测试 AI Agent 是否能在组合优化方面超越传统的启发式算法。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: Manufacturing/FJSP-WF
  • 运行与依赖: README 清晰地记录了独立评估和 frontier_eval 框架的运行命令。它指定了“纯 Python”要求，最大限度地减少了环境配置摩擦。
• 文档质量: 优秀但需润色。对 FJSP-WF 问题的技术解释非常出色。但是，Task.md 中存在冗余标题（例如 ## Instance Data 出现了两次），且 Input / Output Specification 标题存在格式错误。
• 组织结构: 逻辑清晰且具备可扩展性。baseline/、data/ 和 verification/ 的分离符合基准仓库的最佳实践。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未检测到 .env、API 密钥或 IDE 特定文件（如 .vscode/）。result_log.json 对于基准测试的基准线记录是合理的。
• 绝对路径: 未检测到。代码使用 Path(__file__).resolve() 和相对路径来定位数据文件，这对于跨系统执行是健壮的。

[y-ji24]

Review — FJSP-WF (PR #89) — Updated Re-review

Thank you for the timely updates! Of the five previously raised issues, process isolation, test files, baseline run results, and TASK_DETAILS.md have all been correctly fixed. However, a critical implementation issue was identified during re-review.

Correctly fixed items

Process isolation (in-process → subprocess): ✅ Fixed. Added _candidate_runner.py and _run_candidate_subprocess(). The candidate now runs in a separate subprocess with JSON communication, no longer sharing a process with the evaluator.

Test files: ✅ Fixed. Added test_evolve_block.py (4 tests), test_baseline.py (3 tests), and test_evaluator.py (4 tests), totaling 11 tests.

Baseline run results: ✅ Fixed. baseline/result_log.json contains per-instance details for all 30 instances.

TASK_DETAILS.md: ✅ Fixed. The Manufacturing domain and FJSP-WF task entry have been added.

Issue to address

The EVOLVE-BLOCK validation compares the candidate against init.py itself, making the check completely ineffective in the normal evaluation flow

Lines 793-794:

original_solver_path = benchmark_dir / "scripts" / "init.py"
evolve_ok, evolve_msg = _check_evolve_block(candidate_path, original_solver_path)

When running normally (python verification/evaluator.py scripts/init.py):

• candidate_path resolves to scripts/init.py
• original_solver_path also resolves to scripts/init.py
• _check_evolve_block reads the same file twice and compares the two reads

The content of the same file is always equal, so the check always passes. Even if the agent modifies code outside the EVOLVE-BLOCK in init.py (which should be prohibited), _check_evolve_block reads the same modified file as both candidate and original — they are still equal, and the check still passes.

The test test_baseline_fails_evolve_check passes because it passes baseline/solution.py — a file that differs from init.py. But in the real workflow where scripts/init.py is passed, the validation is entirely ineffective. The root cause is that the reference on line 794 is not the frozen baseline, but init.py itself — the very file being modified.

Suggestion: replace original_solver_path on line 794 with the previously resolved baseline_path (line 788), so the validation compares the candidate against the frozen baseline/solution.py. This way, even if the agent modifies init.py's outer code, the validation will detect the difference.

Additional suggestions

1. subprocess.run lacks resource limits: No preexec_fn is set. Consider adding RLIMIT_NPROC, RLIMIT_AS, and RLIMIT_CPU as defense in depth.
2. The parameter name original_path in _check_evolve_block is misleading: The actual value passed is scripts/init.py. Consider renaming it to reference_path to avoid confusion with the semantics of "original copy."

The review can proceed once the above issues are addressed!

Re-review

Thank you for the detailed review and helpful feedback. We have addressed the remaining issues as follows:

✅ Issue 1 — EVOLVE-BLOCK validation

• Updated the reference used for EVOLVE-BLOCK validation from scripts/init.py to references/solver_reference.py, which serves as a frozen reference copy of the original solver.
• Renamed the parameter from original_path to reference_path for better clarity.

✅ Issue 2 — Subprocess resource limits

• Added preexec_fn on non-Windows platforms to apply resource limits:
  • RLIMIT_NPROC (64)
  • RLIMIT_AS (2 GB)
  • RLIMIT_CPU (60 s)

✅ Issue 3 — Parameter naming

• Renamed original_path to reference_path throughout the implementation to better reflect its purpose.

Files Updated

• verification/evaluator.py — updated the EVOLVE-BLOCK reference path, added subprocess resource limits, and renamed the parameter.
• references/solver_reference.py — added as the frozen reference for EVOLVE-BLOCK validation.

Verification

• ✅ All 30 official benchmark instances pass successfully.
• ✅ All 11 verification tests pass.

Appreciate your careful review and believe these changes address the remaining concerns. Thank you again for taking the time to review this benchmark.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new manufacturing benchmark task: FJSP-WF (Flexible Job Shop Scheduling with Worker Flexibility). It is based on the GECCO 2026 FJSSP-WU Competition dataset. The task requires an AI agent to optimize a greedy baseline algorithm (EST+SPT) to minimize the makespan of production schedules, accounting for both machine and worker constraints.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md: Added FJSP-WF to the Manufacturing category registry.
  • benchmarks/Manufacturing/FJSP-WF/README.md: Provided high-level overview, structure, and quick-start commands.
  • benchmarks/Manufacturing/FJSP-WF/Task.md: Detailed engineering background, problem formulation (NP-hard scheduling), I/O specs, and validation rules.
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: Implemented a greedy Earliest Start Time + Shortest Processing Time (EST+SPT) baseline.
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: Recorded baseline performance across 30 official instances.
  • benchmarks/Manufacturing/FJSP-WF/verification/evaluator.py: Logic for schedule validation (precedence, overlaps, eligibility) and scoring.
  • benchmarks/Manufacturing/FJSP-WF/scripts/init.py: The entry point for the agent, containing the editable EVOLVE-BLOCK.
  • benchmarks/Manufacturing/FJSP-WF/data/: Included instance registries, source attribution, and synthetic/official datasets.

2. AI Content Analysis

• Estimated AI Component: 35%
• Reasoning & Evidence:
  • Documentation: The Task.md and README.md follow a highly structured template seen in other frontier_eval tasks. The "Engineering Background" section uses professional, slightly marketing-oriented language ("The economic stakes are high...") typical of LLM-assisted technical writing.
  • Code: The solution.py and evaluator.py use standard Python boilerplate for JSON handling and natural sorting (_natural_key). The logic for the greedy scheduler is standard but clean.
  • Markers: The use of # EVOLVE-BLOCK-START/END markers is a structural requirement of the framework, likely generated or copied from a template.

3. Engineering & Economic Assessment

• Engineering Reality Check: High. This is not a "toy" problem. Flexible Job Shop Scheduling (FJSP) is a classic NP-hard problem. Adding "Worker Flexibility" (WF) reflects modern "Industry 4.0" realities where human skill levels vary. The use of the GECCO 2026 competition dataset ensures the problem scale and complexity are production-grade.
• Economic Value: High. In manufacturing, a 1-5% reduction in makespan directly correlates to increased throughput and reduced operational costs. This benchmark effectively tests an agent's ability to perform combinatorial optimization, a high-value domain in industrial AI.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Manufacturing/FJSP-WF.
  • Execution & Dependencies: The README.md clearly documents the python -m frontier_eval command and local execution via evaluator.py. It specifies that only the Python standard library is allowed, simplifying environment management.
• Documentation Quality: Good. The documentation is comprehensive. However, there are encoding artifacts in Task.md (e.g., 鈥? appearing instead of dashes or special characters), which should be cleaned up for professional presentation.
• Organizational Structure: Excellent. The file organization is modular, separating the baseline, data, scripts, and verification logic. It follows the established repository pattern.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific files (.vscode/) were detected in the diff.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and environment variables (FJSPWF_BENCHMARK_JSON) to handle paths dynamically.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个新的制造业基准测试任务：FJSP-WF（考虑工人灵活性的柔性作业车间调度）。该任务基于 GECCO 2026 FJSSP-WU 竞赛数据集，要求 AI Agent 优化一个贪心基准算法（EST+SPT），在考虑机器和工人双重约束的情况下，最小化生产调度的完工时间（makespan）。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md: 在 Manufacturing 类别中注册了 FJSP-WF。
  • benchmarks/Manufacturing/FJSP-WF/README.md: 提供了高层概述、项目结构和快速启动命令。
  • benchmarks/Manufacturing/FJSP-WF/Task.md: 详细介绍了工程背景、问题定义（NP-hard 调度问题）、输入输出规范和验证规则。
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: 实现了一个贪心算法基准（最早开始时间 + 最短处理时间）。
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: 记录了 30 个官方实例的基准性能数据。
  • benchmarks/Manufacturing/FJSP-WF/verification/evaluator.py: 包含调度验证逻辑（工序先后顺序、重叠检查、资质检查）和评分机制。
  • benchmarks/Manufacturing/FJSP-WF/scripts/init.py: Agent 的入口文件，包含可编辑的 EVOLVE-BLOCK 区域。
  • benchmarks/Manufacturing/FJSP-WF/data/: 包含实例注册表、来源说明以及合成/官方数据集。

2. AI 成分分析

• 预估 AI 含量: 35%
• 判断依据与证据:
  • 文档: Task.md 和 README.md 遵循了该框架高度结构化的模板。其“工程背景”部分使用的专业且略带市场化色彩的语言（如“经济利益巨大……”）是典型的 LLM 辅助技术写作风格。
  • 代码: solution.py 和 evaluator.py 使用了标准的 Python 模板代码进行 JSON 处理和自然排序（_natural_key）。贪心调度器的逻辑标准且整洁。
  • 标记: 使用 # EVOLVE-BLOCK-START/END 标记是框架的结构性要求，很可能是从模板生成或复制的。

3. 工程与经济评估

• 工程现实检验: 高。这并非“玩具”问题。柔性作业车间调度（FJSP）是经典的 NP-hard 问题。加入“工人灵活性”（WF）反映了现代“工业 4.0”中人员技能水平差异的现实。使用 GECCO 2026 竞赛数据集确保了问题的规模和复杂度达到了生产级别。
• 经济价值: 高。在制造业中，完工时间减少 1-5% 直接对应于吞吐量的提高和运营成本的降低。该基准测试有效地测试了 Agent 进行组合优化（工业 AI 的高价值领域）的能力。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: Manufacturing/FJSP-WF。
  • 运行与依赖: README.md 清晰地记录了 python -m frontier_eval 命令和通过 evaluator.py 进行的本地运行方式。明确规定仅允许使用 Python 标准库，简化了环境管理。
• 文档质量: 良好。文档内容详尽。但在 Task.md 中存在编码乱码（例如 鈥? 代替了破折号或特殊字符），应进行清理以保证专业呈现。
• 组织结构: 优秀。文件组织模块化，将基准、数据、脚本和验证逻辑分离，遵循了既有的仓库模式。

5. 安全与隐私检查

• 敏感文件: 未发现异常。Diff 中未检测到 .env、API 密钥或 IDE 配置文件（如 .vscode/）。
• 绝对路径: 未检测到。代码使用 Path(__file__).resolve() 和环境变量（FJSPWF_BENCHMARK_JSON）来动态处理路径。

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark task: FJSP-WF (Flexible Job Shop Scheduling with Worker Flexibility). It is designed to evaluate AI agents' ability to optimize complex manufacturing schedules by assigning operations to machines and workers to minimize makespan.
• Modified File Structure & Modifications:
  • TASK_DETAILS.md: Registered the new FJSP-WF task under the "Manufacturing" category.
  • benchmarks/Manufacturing/FJSP-WF/README.md: Provided a high-level overview, file structure, and quick-start execution commands.
  • benchmarks/Manufacturing/FJSP-WF/Task.md: Detailed technical specification, including problem formulation, constraints (precedence, machine, worker), and scoring logic.
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: Stored baseline performance metrics for 30 official instances.
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: Implemented a greedy baseline algorithm using Earliest Start Time + Shortest Processing Time (EST+SPT).

2. AI Content Analysis

• Estimated AI Component: 25%
• Reasoning & Evidence: The documentation (Task.md and README.md) follows a highly structured, clean template typical of AI-assisted technical writing. The solution.py contains standard boilerplate for path handling (_benchmark_json_path) and natural sorting (_natural_key), which are common AI-generated snippets. However, the domain-specific logic regarding FJSP-WF constraints and the integration with the GECCO 2026 competition dataset indicate significant human engineering and domain expertise.

3. Engineering & Economic Assessment

• Engineering Reality Check: Production-grade. Unlike "toy" scheduling problems, this task incorporates "Worker Flexibility" (varying skill levels and speeds), which is a critical real-world constraint in modern manufacturing. It uses a standardized dataset from a recognized competition (GECCO 2026), ensuring the problem complexity is representative of industrial challenges.
• Economic Value: High. Production scheduling optimization is a multi-billion dollar field. A marginal improvement (e.g., 1%) in makespan directly correlates to increased factory throughput, reduced energy consumption, and optimized labor costs.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Manufacturing/FJSP-WF
  • Execution & Dependencies: The README.md clearly documents the execution commands for both standalone evaluation and the frontier_eval framework. It specifies that only the Python standard library is used, simplifying environment setup.
• Documentation Quality: High. The documentation is comprehensive, covering both the "why" (economic impact) and the "how" (I/O specs).
  • Minor Issue: There is slight redundancy between README.md and Task.md regarding the "How to Run" section, though this aids quick reference.
• Organizational Structure: Logical and Scalable. The separation of baseline/, data/, scripts/, and verification/ follows best practices for benchmark repositories.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were detected. The result_log.json is appropriate for a benchmark baseline.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and environment variables to handle file paths dynamically.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个新的工程基准测试任务：FJSP-WF（带工人柔性的柔性作业车间调度）。旨在评估 AI Agent 通过将工序分配给机器和工人以最小化完工时间（makespan）来优化复杂制造排程的能力。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md: 在“Manufacturing”类别下注册了新的 FJSP-WF 任务。
  • benchmarks/Manufacturing/FJSP-WF/README.md: 提供了高层概述、文件结构和快速启动执行命令。
  • benchmarks/Manufacturing/FJSP-WF/Task.md: 详细的背景技术规范，包括问题公式化、约束条件（前序、机器、工人）和评分逻辑。
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: 存储了 30 个官方实例的基准性能指标。
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: 实现了一个使用最早开始时间 + 最短加工时间 (EST+SPT) 的贪心基准算法。

2. AI 成分分析

• 预估 AI 含量: 25%
• 判断依据与证据: 文档（Task.md 和 README.md）遵循高度结构化、整洁的模板，这是典型的 AI 辅助技术写作风格。solution.py 包含用于路径处理（_benchmark_json_path）和自然排序（_natural_key）的标准样板代码，这些是常见的 AI 生成片段。然而，关于 FJSP-WF 约束的领域特定逻辑以及与 GECCO 2026 竞赛数据集的集成表明了显著的人工工程和领域专业知识。

3. 工程与经济评估

• 工程现实检验: 生产级别。 与“玩具级”调度问题不同，该任务纳入了“工人柔性”（不同的技能水平和速度），这是现代制造业中关键的现实约束。它使用了来自公认竞赛（GECCO 2026）的标准数据集，确保了问题复杂度具有工业挑战的代表性。
• 经济价值: 高。 生产调度优化是一个价值数十亿美元的领域。完工时间（makespan）的边际改进（例如 1%）直接关系到工厂产量的增加、能源消耗的减少以及劳动力成本的优化。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: Manufacturing/FJSP-WF
  • 运行与依赖: README.md 清晰地记录了独立评估和 frontier_eval 框架的执行命令。它明确指出仅使用 Python 标准库，简化了环境安装步骤。
• 文档质量: 高。 文档非常全面，涵盖了“为什么”（经济影响）和“怎么做”（输入/输出规范）。
  • 微小问题: README.md 和 Task.md 在“如何运行”章节存在轻微冗余，但这有助于快速参考。
• 组织结构: 逻辑清晰且具备可扩展性。 baseline/、data/、scripts/ 和 verification/ 的划分符合基准测试仓库的最佳实践。

5. 安全与隐私检查

• 敏感文件: 未发现异常。 未检测到 .env、API 密钥或 IDE 特定配置。result_log.json 对于基准测试的 Baseline 是合适的。
• 绝对路径: 未检测到。 代码使用 Path(__file__).resolve() 和环境变量来动态处理文件路径。

[wrh-human]

Review — FJSP-WF (PR #89) — Third Re-review

Thank you for the continued follow-up! The critical bug from the last review has been correctly fixed, RLIMIT limits have been added, and a frozen reference file has been created. However, one test was not updated to match the new behavior.

---

Issues from previous review — Fixed

EVOLVE-BLOCK validation compared candidate against init.py itself, making the check a no-op ✅

• Created references/solver_reference.py — a frozen, never-modified copy of the original scripts/init.py
• Changed reference path from scripts/init.py to references/solver_reference.py (line 802)
• The candidate's EVOLVE-BLOCK outer regions are now compared against this immutable reference, independent of the candidate's own state

Missing RLIMIT on candidate subprocess ✅

Added preexec_fn with RLIMIT_NPROC=64, RLIMIT_AS=2GB, and RLIMIT_CPU=60, guarded by sys.platform != 'win32'.

Task.md encoding corruption ✅

Fixed garbled Unicode characters and duplicate/ malformed section headings.

---

Issue to address

Test test_baseline_fails_evolve_check was not updated and will now incorrectly fail

The test in test_evolve_block.py passes baseline/solution.py as the candidate and expects the EVOLVE-BLOCK check to fail:

def test_baseline_fails_evolve_check(self):
    proc = _run_evaluator(BASELINE_PATH)
    self.assertNotEqual(proc.returncode, 0)
    self.assertIn("EVOLVE-BLOCK", proc.stderr)

With the new reference path (references/solver_reference.py), both baseline/solution.py and solver_reference.py share the same EVOLVE-BLOCK outer structure (both are frozen copies of the original). The check now passes (A == A), but the test expects failure (assertNotEqual(proc.returncode, 0)). This test will now break.

Either remove this test case (it no longer tests a meaningful scenario, since passing the baseline as a candidate is not a realistic workflow), or change it to verify that SOLVER_PATH (scripts/init.py) passes the EVOLVE-BLOCK check correctly. The remaining tests (test_modified_before_block_fails, test_modified_after_block_fails) remain correct — they modify a copy of init.py and the reference stays unchanged, so the prefix/suffix mismatch is correctly detected.

---

Summary

The critical EVOLVE-BLOCK bug has been correctly fixed, RLIMIT limits have been added, and no new structural issues were found. Fix the test_baseline_fails_evolve_check test and the PR is ready to merge!

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new engineering benchmark task: FJSP-WF (Flexible Job Shop Scheduling with Worker Flexibility). It is designed to evaluate an AI agent's ability to optimize complex manufacturing schedules by assigning operations to machines and workers while minimizing the total completion time (makespan).
• Modified File Structure & Modifications:
  • TASK_DETAILS.md: Added the FJSP-WF task entry under the Manufacturing category.
  • benchmarks/Manufacturing/FJSP-WF/README.md: Created a high-level overview of the benchmark, including its structure, quick-start commands, and data sources.
  • benchmarks/Manufacturing/FJSP-WF/Task.md: Authored a comprehensive task specification covering engineering background, problem formulation, I/O specs, and scoring rules.
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: Provided baseline performance metrics for 30 official instances.
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: Implemented a greedy Earliest Start Time + Shortest Processing Time (EST+SPT) baseline algorithm.
  • benchmarks/Manufacturing/FJSP-WF/scripts/init.py: (Referenced) Created the agent-editable entry point for the optimization task.
  • benchmarks/Manufacturing/FJSP-WF/verification/evaluator.py: (Referenced) Implemented the validation and scoring logic.

2. AI Content Analysis

• Estimated AI Component: 45%
• Reasoning & Evidence: The documentation in Task.md and README.md exhibits a highly structured, "tutorial-like" tone typical of AI-assisted technical writing (e.g., the "Audience and Assumptions" and "Why Worker Flexibility Matters" sections). The solution.py uses standard boilerplate for file path resolution and natural sorting (_natural_key), which is common in AI-generated Python scripts. However, the specific integration with the GECCO 2026 competition dataset and the detailed manufacturing constraints suggest significant human domain expertise and manual configuration.

3. Engineering & Economic Assessment

• Engineering Reality Check: This is a high-fidelity, production-grade problem. Unlike "toy" scheduling puzzles, it incorporates multi-dimensional constraints (machine eligibility, worker skill levels, and job precedence) and uses a dataset from a real-world optimization competition (GECCO 2026). It effectively handles edge cases such as varying worker speeds and machine-specific qualifications.
• Economic Value: High. In industrial engineering, even a 1% reduction in makespan can lead to significant cost savings in energy, labor, and increased throughput. This benchmark provides a rigorous environment to test AI's capability in solving NP-hard combinatorial optimization problems that have direct industrial applications.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Manufacturing/FJSP-WF
  • Execution & Dependencies: The README.md and Task.md provide clear, copy-pasteable CLI commands for running the evaluator and the unified framework. It explicitly states that only the Python standard library is required, simplifying environment setup.
• Documentation Quality: Excellent. The documentation is thorough, providing both the "how" (execution) and the "why" (engineering context). No significant spelling or grammatical errors were detected. The use of Markdown tables for instance families and constraints is highly readable.
• Organizational Structure: The structure is logical and follows the repository's established modular pattern (baseline/, data/, scripts/, verification/), ensuring scalability for adding more instances or variants.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or IDE-specific configurations were found in the diff.
• Absolute Paths: None detected. The code uses Path(__file__).resolve() and environment variables to handle file paths dynamically.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个新的工程基准测试任务：FJSP-WF（具有工人灵活性的柔性作业车间调度）。该任务旨在评估 AI Agent 在最小化总完工时间（makespan）的目标下，通过优化机器和工人的分配来处理复杂制造调度问题的能力。
• 修改的文件结构与变更摘要:
  • TASK_DETAILS.md: 在 Manufacturing 类别下添加了 FJSP-WF 任务条目。
  • benchmarks/Manufacturing/FJSP-WF/README.md: 创建了基准测试的高层概述，包括结构、快速启动命令和数据来源。
  • benchmarks/Manufacturing/FJSP-WF/Task.md: 编写了详尽的任务规范，涵盖工程背景、问题公式化、输入/输出规范和评分规则。
  • benchmarks/Manufacturing/FJSP-WF/baseline/result_log.json: 提供了 30 个官方实例的基准性能指标。
  • benchmarks/Manufacturing/FJSP-WF/baseline/solution.py: 实现了一个基于贪心策略（最早开始时间 + 最短处理时间，EST+SPT）的基准算法。
  • benchmarks/Manufacturing/FJSP-WF/scripts/init.py: (引用) 创建了供 Agent 编辑的优化任务入口文件。
  • benchmarks/Manufacturing/FJSP-WF/verification/evaluator.py: (引用) 实现了验证和评分逻辑。

2. AI 成分分析

• 预估 AI 含量: 45%
• 判断依据与证据: Task.md 和 README.md 中的文档表现出高度结构化、类似于“教程”的语气，这是 AI 辅助技术写作的典型特征（例如“受众与假设”和“为什么工人灵活性很重要”章节）。solution.py 使用了标准的文件路径解析和自然排序模板（_natural_key），这在 AI 生成的 Python 脚本中非常常见。然而，与 GECCO 2026 竞赛数据集的具体集成以及详细的制造约束说明，表明存在显著的人工领域专业知识和手动配置。

3. 工程与经济评估

• 工程现实检验: 这是一个高保真、生产级别的工程问题。与“玩具级”调度谜题不同，它整合了多维约束（机器适用性、工人技能水平和工序优先级），并使用了来自真实世界优化竞赛（GECCO 2026）的数据集。它有效地处理了诸如工人速度差异和机器特定资质等边缘情况。
• 经济价值: 高。在工业工程中，即使完工时间减少 1%，也能在能源、劳动力和提高产量方面节省大量成本。该基准测试提供了一个严格的环境，用于测试 AI 解决具有直接工业应用背景的 NP-hard 组合优化问题的能力。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: Manufacturing/FJSP-WF
  • 运行与依赖: README.md 和 Task.md 提供了清晰、可直接复制的 CLI 命令，用于运行评估器和统一框架。文档明确指出仅需 Python 标准库，简化了环境配置。
• 文档质量: 优秀。文档非常详尽，既提供了“如何做”（执行），也提供了“为什么”（工程背景）。未发现明显的拼写或语法错误。使用 Markdown 表格展示实例族和约束条件，极具可读性。
• 组织结构: 文件组织逻辑清晰，遵循仓库既有的模块化模式（baseline/, data/, scripts/, verification/），确保了添加更多实例或变体时的可扩展性。

5. 安全与隐私检查

• 敏感文件: 未发现异常。Diff 中未发现 .env、API 密钥或 IDE 特定配置。
• 绝对路径: 未检测到。代码使用 Path(__file__).resolve() 和环境变量来动态处理文件路径。

[y-ji24]

Review — FJSP-WF (PR #89) — Third Re-review

Thank you for the continued follow-up! The critical bug from the last review has been correctly fixed, RLIMIT limits have been added, and a frozen reference file has been created. However, one test was not updated to match the new behavior.

Issues from previous review — Fixed

EVOLVE-BLOCK validation compared candidate against init.py itself, making the check a no-op ✅

• Created references/solver_reference.py — a frozen, never-modified copy of the original scripts/init.py
• Changed reference path from scripts/init.py to references/solver_reference.py (line 802)
• The candidate's EVOLVE-BLOCK outer regions are now compared against this immutable reference, independent of the candidate's own state

Missing RLIMIT on candidate subprocess ✅

Added preexec_fn with RLIMIT_NPROC=64, RLIMIT_AS=2GB, and RLIMIT_CPU=60, guarded by sys.platform != 'win32'.

Task.md encoding corruption ✅

Fixed garbled Unicode characters and duplicate/ malformed section headings.

Issue to address

Test test_baseline_fails_evolve_check was not updated and will now incorrectly fail

The test in test_evolve_block.py passes baseline/solution.py as the candidate and expects the EVOLVE-BLOCK check to fail:

def test_baseline_fails_evolve_check(self):
    proc = _run_evaluator(BASELINE_PATH)
    self.assertNotEqual(proc.returncode, 0)
    self.assertIn("EVOLVE-BLOCK", proc.stderr)

With the new reference path (references/solver_reference.py), both baseline/solution.py and solver_reference.py share the same EVOLVE-BLOCK outer structure (both are frozen copies of the original). The check now passes (A == A), but the test expects failure (assertNotEqual(proc.returncode, 0)). This test will now break.

Either remove this test case (it no longer tests a meaningful scenario, since passing the baseline as a candidate is not a realistic workflow), or change it to verify that SOLVER_PATH (scripts/init.py) passes the EVOLVE-BLOCK check correctly. The remaining tests (test_modified_before_block_fails, test_modified_after_block_fails) remain correct — they modify a copy of init.py and the reference stays unchanged, so the prefix/suffix mismatch is correctly detected.

Summary

The critical EVOLVE-BLOCK bug has been correctly fixed, RLIMIT limits have been added, and no new structural issues were found. Fix the test_baseline_fails_evolve_check test and the PR is ready to merge!

Re-review

Thank you for the continued review and feedback. The remaining test issue has now been resolved.

Issue — Outdated EVOLVE-BLOCK test: ✅ Fixed

• Removed the obsolete test_baseline_fails_evolve_check test case.
• Removed the unused BASELINE_PATH constant from test_evolve_block.py.
• This test was no longer appropriate because baseline/solution.py and references/solver_reference.py now intentionally share the same protected EVOLVE-BLOCK structure.
• The remaining EVOLVE-BLOCK tests continue to provide full coverage of the intended protection mechanism:
  • test_solver_passes_evolve_check
  • test_modified_before_block_fails
  • test_modified_after_block_fails

Verification

• ✅ All 10/10 tests pass.
• ✅ All 30 official benchmark instances pass with both the baseline and solver implementations.
• ✅ No benchmark behavior or evaluation logic was changed as part of this cleanup.

Thanks again for the careful review.

[wrh-human]

Review — FJSP-WF (PR #89) — Final Review

Thank you for the continued follow-up! The test issue identified in the last round has been correctly fixed. After a line-by-line review, all previously raised issues are now resolved.

---

Final status of all issues across three review rounds

• EVOLVE-BLOCK restructuring ✅ — init.py now has a clean separation: read-only prefix (parsers), editable region (placement algorithm), read-only suffix (output writer, CLI)
• EVOLVE-BLOCK boundary validation ✅ — Compares against references/solver_reference.py (frozen copy), not against init.py itself
• Subprocess isolation ✅ — Candidate runs in a subprocess via _candidate_runner.py
• RLIMIT resource limits ✅ — RLIMIT_NPROC=64, RLIMIT_AS=2GB, RLIMIT_CPU=60 added
• solver_reference.py ✅ — Frozen copy of the original init.py, used as the EVOLVE-BLOCK validation reference
• TASK_DETAILS.md ✅ — Manufacturing domain and FJSP-WF task entry added
• Task.md encoding fixes ✅ — Garbled Unicode characters and heading formatting corrected
• Test files ✅ — 11 tests covering EVOLVE-BLOCK boundaries, baseline, and evaluator pipeline
• Baseline run results ✅ — baseline/result_log.json with per-instance details for all 30 instances
• Random seed ✅ — random.seed(42) added
• Outdated test removed ✅ — test_baseline_fails_evolve_check deleted

New issues found: None

All issues have been resolved. No new issues were identified. Ready to merge.