Add FPGA Placement Optimization benchmark

[y-ji24]

Summary

This PR introduces the FPGA Placement Optimization benchmark.

The benchmark evaluates an agent's ability to design and improve placement algorithms through an editable artifact (scripts/init.py).

Design

• Editable artifact:
  scripts/init.py

• Baseline:
  reference only, not modified by agents

• Evaluation:
  candidate-generated solution.pl is evaluated independently

Benchmark Features

• Bookshelf format parsing
• Placement legality verification
• HPWL-based scoring
• Multiple benchmark support

Dataset

The default benchmark is bundled.

ISPD 2016 benchmarks are supported as optional external datasets.

Validation

Tested:

• default benchmark evaluation
• ISPD benchmark loading
• legality checks
• invalid benchmark handling

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new evaluation metric ("Medal Score"), a lightweight benchmark subset (v1-lite), and comprehensive design/validation documentation for a new FPGA placement optimization task.
• Modified File Structure & Modifications:
  • .gitignore: Whitelisted leaderboard/*.csv and the new v1_lite.yaml configuration.
  • README.md & README_zh-CN.md: Updated with "News" regarding the Medal Score and v1-lite. Updated the leaderboard table to reflect these new metrics and future-dated (2026) project milestones.
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_DesignValidation.md: A new, highly detailed audit of the aug-elfPlace baseline, identifying discrepancies between the "Clock-Aware" naming and actual code implementation (e.g., dead code in C++ parsers).
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_Design_Report.md: A new architectural blueprint for the FPGA placement task, outlining scoring logic (HPWL-based), validation gates, and implementation roadmaps.

2. AI Content Analysis

• Estimated AI Component: 25%
• Reasoning & Evidence: The README updates and leaderboard data are clearly manual entries. The two new Markdown reports (DesignValidation.md and Design_Report.md) exhibit a highly structured, LLM-like formatting (tables, clear headers, pros/cons lists). However, the content is extremely domain-specific and references internal code locations (e.g., PlaceDB.cpp:805-813, lut_ff_legalization.py:240). This suggests an engineer used an AI to structure and draft findings based on a very thorough manual code audit.

3. Engineering & Economic Assessment

• Engineering Reality Check:
  • High Reality: The "Medal Score" addresses a sophisticated benchmarking problem: how to aggregate performance across diverse tasks without letting "long-tail" marginal gains in easy tasks skew the average.
  • Critical Audit: The DesignValidation.md is a standout piece of engineering work. It correctly identifies that the baseline lacks actual clock-aware optimization despite the naming, preventing the creation of a "toy" benchmark based on false assumptions.
• Economic Value: High.
  • Efficiency: The v1-lite subset significantly reduces the compute cost and time for developer iterations.
  • Technical Debt: The FPGA audit prevents the implementation of a flawed benchmark, saving future engineering hours that would have been spent debugging non-functional "clock" constraints.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: v1-lite (as a batch configuration) and ClockAwarePlacement (proposed in design docs).
  • Execution & Dependencies: The README explicitly provides the command to run the new v1-lite subset (frontier_eval/conf/batch/v1_lite.yaml). The design report correctly identifies the need for a pure-Python evaluator to avoid heavy C++/CUDA dependencies.
• Documentation Quality: Excellent. The design reports are thorough, and the README updates are clear. Note: The dates are set in June 2026, which appears to be a project roadmap convention rather than a typo.
• Organizational Structure: Logical. The design and validation documents are co-located with the specific benchmark they describe.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, .vscode, or cache files were included.
• Absolute Paths: None detected. The design report explicitly recommends using relative paths and pure-Python scoring to maintain portability.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了新的评分指标（“勋章分” Medal Score）、轻量级评测子集（v1-lite），以及针对新 FPGA 布局优化任务的详细设计与验证文档。
• 修改的文件结构与变更摘要:
  • .gitignore: 将 leaderboard/*.csv 和新的 v1_lite.yaml 配置列入白名单。
  • README.md & README_zh-CN.md: 更新了关于 Medal Score 和 v1-lite 的“新闻”动态。更新了排行榜表格以反映这些新指标，并标注了 2026 年的项目里程碑。
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_DesignValidation.md: 一份全新的、极其详尽的 aug-elfPlace 基准线审计文档，指出了“时钟感知”命名与实际代码实现之间的差异（例如 C++ 解析器中的死代码）。
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_Design_Report.md: FPGA 布局任务的新架构蓝图，概述了评分逻辑（基于 HPWL）、验证门槛和实施路线图。

2. AI 成分分析

• 预估 AI 含量: 25%
• 判断依据与证据: README 的更新和排行榜数据显然是手动录入的。两份新的 Markdown 报告（DesignValidation.md 和 Design_Report.md）表现出高度结构化、类似 LLM 的格式（表格、清晰的标题、优缺点列表）。然而，内容具有极强的领域专业性，并引用了内部代码位置（如 PlaceDB.cpp:805-813, lut_ff_legalization.py:240）。这表明工程师在进行了非常彻底的手动代码审计后，利用 AI 进行了结构化整理和草拟。

3. 工程与经济评估

• 工程现实检验:
  • 高度现实: “勋章分”解决了基准测试中的一个复杂问题：如何在不让简单任务中的“长尾”边际收益扭曲平均值的情况下，汇总不同任务的性能。
  • 批判性审计: DesignValidation.md 是工程实践中的亮点。它准确识别出基准线虽然命名为“时钟感知”，但实际上缺乏相关优化逻辑，从而避免了基于错误假设构建“玩具级”基准测试。
• 经济价值: 高。
  • 效率: v1-lite 子集显著降低了开发人员迭代的计算成本和时间。
  • 技术债务: FPGA 审计防止了缺陷基准测试的实施，节省了未来用于调试无效“时钟”约束的工程时间。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: v1-lite（作为批处理配置）和 ClockAwarePlacement（在设计文档中提出）。
  • 运行与依赖: README 明确提供了运行新 v1-lite 子集的命令（frontier_eval/conf/batch/v1_lite.yaml）。设计报告正确识别了对纯 Python 评估器的需求，以避免沉重的 C++/CUDA 依赖。
• 文档质量: 优秀。设计报告非常详尽，README 更新清晰。注：日期设定在 2026 年 6 月，这似乎是项目路线图的惯例而非拼写错误。
• 组织结构: 符合逻辑。设计和验证文档与它们所描述的特定基准测试存放在一起。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未包含 .env、.vscode 或缓存文件。
• 绝对路径: 未检测到。设计报告明确建议使用相对路径和纯 Python 评分以保持可移植性。

[wrh-human]

Review — FPGAPlacementOptimization (PR #85)

Thank you for contributing this benchmark! FPGA placement optimization is a genuine EDA engineering problem, the HPWL scoring and three legality gates are well-designed, and the canonical reference implementation ensures evaluator independence. After a line-by-line review of all code and files, one issue was identified that needs to be addressed before merging.

---

Evaluation by dimension

1. Domain, Economic Value, and Frontier-Eng Fit ✅

FPGA placement optimization is a real electronic design automation (EDA) problem — logic cells (LUTs, FFs, DSPs, BRAMs) must be assigned to legal sites on the FPGA grid to minimize wirelength. Placement quality directly affects chip area, power consumption, and timing closure. The task uses the ISPD 2016 FPGA Placement Contest Bookshelf format, and HPWL is the industry-standard metric for placement quality. Economic value is clear.

2. Not purely numerical ✅

The agent modifies the placement algorithm logic in scripts/init.py (row-scan → heuristic/analytical/simulated annealing approaches), not numerical parameters.

3. Search space ✅

FPGA placement is an NP-hard combinatorial optimization problem. The example design has 3,336 instances (72 fixed IO, 3,264 movable). Brute-force search is infeasible.

4. Evaluator and engineering verification ✅

The evaluator runs the candidate in a subprocess (subprocess.run), then independently parses solution.pl and verifies legality and computes HPWL via canonical.py. All three legality gates are independently implemented in the canonical reference, independent of the candidate and not trusting candidate-reported values.

5. Constraint enforcement ✅

All three gates are independently implemented in check_site_type_compatibility(), check_site_capacity(), and check_carry_chain_integrity(). Violating any single gate returns valid=0.0.

6. Baseline experiment ✅

baseline/result_log.txt is included, documenting the fpga-example1 run (HPWL=210,721, all gates pass, runtime 0.19s).

7. Scoring system ✅

combined_score = -HPWL. HPWL is the industry-standard metric for placement quality. The scoring directly uses the raw wirelength — no clipping, no normalization. Discriminative power is clear and well-supported.

---

Issue to address

Issue 1: EVOLVE-BLOCK wraps the entire file — no read-only shell

In scripts/init.py, EVOLVE-BLOCK-START is at line 1 and EVOLVE-BLOCK-END is at line 314, wrapping the entire file — including imports, the main() function, argument parsing, and Bookshelf file I/O. There is no read-only shell. The evaluator contains no EVOLVE-related validation logic and never checks whether the candidate modified code outside the EVOLVE-BLOCK markers.

The "DO NOT MODIFY" comments inside the file have no machine-enforceable mechanism — if the agent modifies the CLI interface or the solution.pl output format, the evaluator may be unable to correctly parse the candidate's output.

Suggestion: move the read-only CLI interface, argument parsing, and Bookshelf file I/O logic outside the EVOLVE-BLOCK. Restrict the EVOLVE-BLOCK to only wrap the placement algorithm function, and add EVOLVE-BLOCK boundary validation to the evaluator.

---

FPGA placement optimization has the right direction, and the HPWL scoring and legality gates are well-designed. The review can proceed once the above issue is addressed.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new scoring methodology (Medal Score), a reduced benchmark subset (v1-lite), and comprehensive design/validation documentation for a new FPGA placement optimization task. It aims to make the benchmark more robust against noise and more accessible for rapid iteration.
• Modified File Structure & Modifications:
  • .gitignore: Updated to allow leaderboard CSVs and the new v1_lite.yaml config, while ignoring solution.pl (FPGA output).
  • README.md & README_zh-CN.md: Added "News" section, introduced the Medal Score (Gold/Silver/Bronze) logic, documented the v1-lite subset, and updated the leaderboard with new metrics and 2026-04-14 snapshot data.
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_DesignValidation.md: (New) A technical audit of the aug-elfPlace baseline, identifying dead code and missing clock-aware features in the original repository.
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_Design_Report.md: (New) A detailed architectural blueprint for the FPGA placement task, proposing an output-based scoring method to avoid heavy C++ dependencies.

2. AI Content Analysis

• Estimated AI Component: 15%
• Reasoning & Evidence: The documentation (Design Validation and Design Report) is highly specific and reflects a manual audit of a C++ and Python codebase. The references to specific line numbers (e.g., PlaceDB.cpp:805-813, LibCell.h:107-109) and the identification of "dead code" in Pybind11 bindings suggest human-led technical research. AI may have been used to structure the Markdown or translate the README updates, but the core technical findings are domain-specific and evidence-based.

3. Engineering & Economic Assessment

• Engineering Reality Check:
  • The Medal Score addresses a common pitfall in LLM benchmarking: "long-tail" noise where tiny numerical improvements skew rankings. By freezing top-3 baselines, it creates a stable "podium" system.
  • The v1-lite subset (10 tasks) is a pragmatic response to the high cost/time of running the full 47-task suite, enabling faster developer iteration.
  • The FPGA Design Validation shows high engineering maturity by identifying that the "Clock-Aware" label in the baseline was aspirational rather than functional, preventing the team from building on a false premise.
• Economic Value: High. It reduces the "cost per evaluation" (via v1-lite) and increases the "signal-to-noise ratio" of the results (via Medal Score), making the benchmark more valuable for commercial LLM providers and researchers.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: v1-lite (batch configuration).
  • Execution & Dependencies: The README clearly documents the command to run the lite version: frontier_eval/conf/batch/v1_lite.yaml. The FPGA design report explicitly advocates for a "Pure-Python" evaluator to eliminate C++ compilation risks.
• Documentation Quality: Excellent. The design reports are thorough, covering repository patterns, baseline investigation, and risk review. No significant grammatical errors or formatting inconsistencies were detected.
• Organizational Structure: Logical. The design documents are correctly placed within the specific benchmark sub-directory, following the project's modular structure.

5. Security & Privacy Check

• Sensitive Files: Clean. The .gitignore was properly updated to prevent the accidental commitment of solution files (*.pl).
• Absolute Paths: None detected. All references use relative paths or environment variables like {python}.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了全新的评分机制（Medal Score 奖牌分）、精简的评测子集（v1-lite），以及针对新增 FPGA 布局优化任务的详尽设计与验证文档。旨在提高基准测试的抗噪性并加速开发迭代。
• 修改的文件结构与变更摘要:
  • .gitignore: 更新以允许提交排行榜 CSV 和新的 v1_lite.yaml 配置，同时忽略 solution.pl（FPGA 输出文件）。
  • README.md & README_zh-CN.md: 新增“新闻”栏目，引入金/银/铜奖牌分逻辑，记录 v1-lite 子集信息，并根据 2026-04-14 的快照数据更新了排行榜指标。
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_DesignValidation.md: （新增） 对 aug-elfPlace 基准代码的技术审计，识别了原仓库中的死代码和缺失的时钟感知功能。
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_Design_Report.md: （新增） FPGA 布局任务的详细架构蓝图，提议采用基于输出的评分方法以规避复杂的 C++ 环境依赖。

2. AI 成分分析

• 预估 AI 含量: 15%
• 判断依据与证据: 文档（设计验证和设计报告）具有极高的针对性，反映了对 C++ 和 Python 代码库的人工审计。对特定行号的引用（如 PlaceDB.cpp:805-813）以及对 Pybind11 绑定中“死代码”的识别，表明这是由人类工程师主导的技术研究。AI 可能被用于辅助 Markdown 结构化或 README 的翻译，但核心技术发现是基于特定领域的实证研究。

3. 工程与经济评估

• 工程现实检验:
  • 奖牌分 (Medal Score) 解决了 LLM 基准测试中的常见问题：即微小的数值提升（长尾噪声）干扰排名。通过冻结前三名基准，建立了一个稳定的“领奖台”体系。
  • v1-lite 子集（10 个任务）是对运行全量 47 个任务的高昂成本/时间的务实回应，支持更快的开发迭代。
  • FPGA 设计验证 显示了极高的工程成熟度，识别出基准代码中的“时钟感知”标签仅是愿景而非功能，避免了基于错误前提进行开发。
• 经济价值: 高。通过 v1-lite 降低了“单次评测成本”，并通过奖牌分提高了结果的“信噪比”，使其对商业 LLM 供应商和研究机构更具价值。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: v1-lite (批处理配置)。
  • 运行与依赖: README 清晰记录了运行 lite 版本的命令：frontier_eval/conf/batch/v1_lite.yaml。FPGA 设计报告明确主张使用“纯 Python”评估器，以消除 C++ 编译风险。
• 文档质量: 优秀。设计报告非常详尽，涵盖了仓库模式、基准调查和风险审查。未发现明显的语法错误或格式不一致。
• 组织结构: 符合逻辑。设计文档正确放置在特定的基准子目录中，遵循项目的模块化结构。

5. 安全与隐私检查

• 敏感文件: 未发现异常。.gitignore 已正确更新，防止意外提交解决方案文件 (*.pl)。
• 绝对路径: 未检测到。所有引用均使用相对路径或环境变量（如 {python}）。

[y-ji24]

Review — FPGAPlacementOptimization (PR #85)

Thank you for contributing this benchmark! FPGA placement optimization is a genuine EDA engineering problem, the HPWL scoring and three legality gates are well-designed, and the canonical reference implementation ensures evaluator independence. After a line-by-line review of all code and files, one issue was identified that needs to be addressed before merging.

Evaluation by dimension

1. Domain, Economic Value, and Frontier-Eng Fit ✅

FPGA placement optimization is a real electronic design automation (EDA) problem — logic cells (LUTs, FFs, DSPs, BRAMs) must be assigned to legal sites on the FPGA grid to minimize wirelength. Placement quality directly affects chip area, power consumption, and timing closure. The task uses the ISPD 2016 FPGA Placement Contest Bookshelf format, and HPWL is the industry-standard metric for placement quality. Economic value is clear.

2. Not purely numerical ✅

The agent modifies the placement algorithm logic in scripts/init.py (row-scan → heuristic/analytical/simulated annealing approaches), not numerical parameters.

3. Search space ✅

FPGA placement is an NP-hard combinatorial optimization problem. The example design has 3,336 instances (72 fixed IO, 3,264 movable). Brute-force search is infeasible.

4. Evaluator and engineering verification ✅

The evaluator runs the candidate in a subprocess (subprocess.run), then independently parses solution.pl and verifies legality and computes HPWL via canonical.py. All three legality gates are independently implemented in the canonical reference, independent of the candidate and not trusting candidate-reported values.

5. Constraint enforcement ✅

All three gates are independently implemented in check_site_type_compatibility(), check_site_capacity(), and check_carry_chain_integrity(). Violating any single gate returns valid=0.0.

6. Baseline experiment ✅

baseline/result_log.txt is included, documenting the fpga-example1 run (HPWL=210,721, all gates pass, runtime 0.19s).

7. Scoring system ✅

combined_score = -HPWL. HPWL is the industry-standard metric for placement quality. The scoring directly uses the raw wirelength — no clipping, no normalization. Discriminative power is clear and well-supported.

Issue to address

Issue 1: EVOLVE-BLOCK wraps the entire file — no read-only shell

In scripts/init.py, EVOLVE-BLOCK-START is at line 1 and EVOLVE-BLOCK-END is at line 314, wrapping the entire file — including imports, the main() function, argument parsing, and Bookshelf file I/O. There is no read-only shell. The evaluator contains no EVOLVE-related validation logic and never checks whether the candidate modified code outside the EVOLVE-BLOCK markers.

The "DO NOT MODIFY" comments inside the file have no machine-enforceable mechanism — if the agent modifies the CLI interface or the solution.pl output format, the evaluator may be unable to correctly parse the candidate's output.

Suggestion: move the read-only CLI interface, argument parsing, and Bookshelf file I/O logic outside the EVOLVE-BLOCK. Restrict the EVOLVE-BLOCK to only wrap the placement algorithm function, and add EVOLVE-BLOCK boundary validation to the evaluator.

FPGA placement optimization has the right direction, and the HPWL scoring and legality gates are well-designed. The review can proceed once the above issue is addressed.

Re-review

Thank you again for the careful review and helpful feedback. We have addressed the remaining issue as follows:

✅ Issue 1 — EVOLVE-BLOCK boundary

• Restricted the EVOLVE-BLOCK to lines 119–248, covering only the placement algorithm (row_scan_place) and its helper functions.
• Moved the CLI interface, argument parsing, and Bookshelf I/O logic (write_pl, main, parse_nodes, parse_pl, and parse_scl) outside the editable region.
• Clearly marked the protected section as:
  READ-ONLY: Output Writer and CLI Entry Point (frozen -- do not modify)
• Added validate_evolve_boundary() to the evaluator. It verifies at runtime that all protected functions remain outside the EVOLVE-BLOCK. Any modification to the frozen code will cause the candidate to be marked as invalid.

Verification

• ✅ Evaluator passes successfully.
• ✅ HPWL = 210721.
• ✅ All three validation gates pass.
• ✅ Final result: valid = 1.0.

Thank you again for the detailed review. We believe these changes address the concern regarding the EVOLVE-BLOCK boundary.

[wrh-human]

Review — FPGAPlacementOptimization (PR #85) — Updated Re-review

Thank you for the prompt revisions. The only previously raised issue has been fully addressed, and after a line-by-line review of all changed code, no new issues were found.

---

Previously raised issue — Fixed

EVOLVE-BLOCK wrapped the entire file with no read-only shell ✅

scripts/init.py has been correctly restructured into a three-part layout:

• Before EVOLVE-BLOCK-START (read-only prefix, lines 33-118): parse_nodes(), parse_pl(), parse_scl() — benchmark parsing functions
• Inside EVOLVE-BLOCK (editable, lines 119-248): is_slice_cell(), get_site_type(), row_scan_place() — placement algorithm
• After EVOLVE-BLOCK-END (read-only suffix, lines 257-319): write_pl(), main() — output writer and CLI entry point

The evaluator now includes a validate_evolve_boundary() function (line 63) that checks whether protected functions are inside the EVOLVE-BLOCK before running the candidate. The _PROTECTED_FUNCTIONS list includes parse_nodes, parse_pl, parse_scl, write_pl, and main(), precisely matching the read-only functions in init.py.

Task.md has been updated with EVOLVE-BLOCK boundary documentation. The "aug-elfPlace" baseline directory has been cleaned up.

---

Other changes verified

• Candidate subprocess timeout increased from 300 to 3600 seconds, suitable for large ISPD designs
• Removed unused RESOURCES parsing logic in parse_scl()
• row_scan_place() now has a complete docstring and type annotations
• Overall code style and comment quality have been improved

---

New issues found: None

The only issue from the previous review has been fully addressed, and no new issues or remaining defects were identified. Ready to merge!