arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.19787cs.AI
收藏

ORAgentBench: LLM代理能否解决具有挑战性的端到端运筹学任务?

ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

Southeast University(东南大学);Waseda University(早稻田大学);Nanyang Technological University(南洋理工大学)

Jiajun Li, Mingshu Cai, Yixuan Li, Yu Ding, Ran Hou, Guanyu Nie, Xiongwei Han, Wanyuan Wang

AI总结:

提出ORAgentBench基准,评估LLM代理在端到端运筹学任务中的表现,发现当前代理通过率仅35.51%,主要受策略性弱点限制。

摘要:

大型语言模型越来越多地被部署为可执行环境中多步任务的自主代理,但它们执行现实运筹学工作的能力仍不明确。现有的运筹学评估通常将建模与求解分离,依赖预形式化或纯文本实例,很少测试从操作工件到验证决策的完整工作流程。在这项工作中,我们引入了ORAgentBench,一个基于执行环境的基准,用于评估自主代理在具有挑战性的端到端运筹学任务上的表现。它包含107个经过人工审核的任务,涵盖多样化的操作场景,每个任务都打包在一个隔离环境中,包含自然语言简介、多文件数据、配置工件和所需的提交模式。代理必须编写并运行解决方案代码,其提交由隐藏验证器根据模式有效性、硬约束可行性和归一化目标质量进行评估。对十四个前沿代理模型配置的实验表明,当前代理远未达到可靠的运筹学实践。最佳代理仅通过35.51%的所有任务和20.59%的困难任务,许多可行的提交仍低于所需的质量阈值。失败分析进一步表明,错误主要由策略性弱点主导,包括遗漏操作规则、脆弱的公式化、弱可行解构造以及解改进不足。运筹学特定的程序性技能增加了困难任务的可行性,但并未可靠地提高解质量或通过率。这些结果表明,运筹学代理的进展需要超越合理的优化代码,转向可靠、高质量的操作决策。

英文摘要:

Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In this work, we introduce ORAgentBench, an execution-grounded benchmark for evaluating autonomous agents on challenging end-to-end operations research tasks. It contains 107 human-reviewed tasks across diverse operational scenarios, each packaged in an isolated environment with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents must write and run solution code, and their submissions are evaluated by hidden validators for schema validity, hard-constraint feasibility, and normalized objective quality. Experiments with fourteen frontier agent-model configurations show that current agents remain far from reliable OR practice. The best agent passes only 35.51% of all tasks and 20.59% of hard tasks, and many feasible submissions still fall below the required quality threshold. Failure analysis further shows that errors are dominated by strategic weaknesses, including missed operational rules, brittle formulations, weak feasible-solution construction, and insufficient solution improvement. OR-specific procedural skills increase hard-task feasibility, but do not reliably improve solution quality or pass rate. These results suggest that progress in OR agents requires moving beyond plausible optimization code toward dependable, high-quality operational decision-making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑