Frontier-Eng：基于生成优化的现实工程任务中自演化代理的基准测试
Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization
AI总结：
Frontier-Eng通过工业级模拟器和验证器，评估生成优化中代理在现实工程任务中的表现，发现GPT 5.4表现最稳健，但所有模型仍面临挑战，分析显示改进频率和幅度呈双幂律衰减。
摘要：
当前LLM代理基准测试主要关注二元通过/失败任务，如代码生成或基于搜索的问题回答，往往忽视现实工程中通过可行设计迭代优化的价值。为此，我们引入Frontier-Eng，一个经人类验证的生成优化基准测试——一种迭代的提出-执行-评估循环，代理生成候选物，接收可执行验证器反馈，并在固定交互预算下修订。该基准测试涵盖5个广泛工程类别中的47个任务。与以往的套件不同，Frontier-Eng任务基于工业级模拟器和验证器，提供连续奖励信号并在受限预算下强制执行硬可行性约束。我们评估了八种前沿语言模型，使用代表性搜索框架，发现尽管GPT 5.4表现最稳健，但基准测试仍对所有模型具有挑战性。我们的分析表明，改进频率和幅度呈双幂律衰减（~1/迭代和~1/改进次数）。我们进一步表明，尽管宽度提高并行性和多样性，深度在固定预算下仍对硬获改进至关重要。Frontier-Eng为评估AI代理整合领域知识与可执行反馈解决复杂、开放性工程问题的能力建立了新标准。
英文摘要：
Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning $47$ tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency ($\sim$ 1/iteration) and magnitude ($\sim$ 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
