OptiArena：在固定资源预算下，大型语言模型能否改进可执行算法？
OptiArena: Can LLMs Improve Executable Algorithms under Fixed Resource Budgets?
浏览论文内容
中文总结 AI 辅助
OptiArena是一个固定资源预算下的测试平台，通过五轮代码编辑评估LLM改进可执行游戏算法的能力，发现模型改进弱起点比优化胜任基线更一致。
中文摘要 AI 辅助
静态问答和代码生成基准仅部分反映了大型语言模型（LLM）作为编码智能体和研究工具所扮演的角色。我们引入了OptiArena，一个受预算控制的测试平台，用于研究LLM是否能够在固定的最小脚手架内、在有界评估器反馈和固定资源预算下，通过五轮代码编辑改进可执行的游戏算法。该测试平台采用两种优化机制、表面混淆控制、校准参考、保留/压力分割，以及针对性能退化和异常失败的诊断，LLM API成本与本地评估器墙钟时间分开报告。实证研究提出三个问题：模型能否缩小指定的弱起点与可编辑的胜任基线之间的校准差距，能否在不损害可编辑胜任基线的情况下对其进行改进，以及改进能否在表面混淆控制下保持。在十二个前沿LLM和五个游戏中，模型改进指定弱起点的表现比改进可编辑胜任基线更一致，但不同游戏和模型之间存在显著差异。OptiArena为在此研究的五轮编辑、固定脚手架设置中衡量有界资源算法优化提供了一个实用测试平台。代码可在以下网址获取：https://this https URL。
英文摘要
Static QA and code-generation benchmarks only partially capture the role that large language models (LLMs) now play as coding agents and research tools. We introduce OptiArena, a budget-controlled testbed for studying whether LLMs can improve executable game-playing algorithms through five rounds of code edits within a fixed minimal scaffold and under bounded evaluator feedback and fixed resource budgets. The testbed uses two optimization regimes, surface obfuscation controls, calibrated references, held-out/stress splits, and diagnostics for degradation and exceptional failures, with LLM API cost reported separately from local evaluator wall-clock. The empirical study asks three questions: whether models can close the calibrated gap between a designated weak starter and an editable competent baseline, whether they can refine editable competent baselines without damaging them, and whether gains survive surface obfuscation controls. Across twelve frontier LLMs and five games, models improve designated weak starters more consistently than they refine editable competent baselines, with substantial variation across games and models. OptiArena provides a practical testbed for measuring bounded-resource algorithm optimization within the five-edit, fixed-scaffold setting studied here. Code is available at https://github.com/WJ-Peng/OptiArena.
发表机构
- Adelaide University（阿德莱德大学）
机构由 AI 辅助整理，请以论文原文为准。
