发表机构
University of Rochester; RamenVR（罗切斯特大学; RamenVR）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对游戏引擎编码智能体评估，提出CraftBenchUE确定性框架，含70任务，发现C++完成率显著高于蓝图，部分蓝图提交虽过资产检查但未过运行时测试。
AI 中文摘要
在游戏引擎中构建游戏功能需要的不仅仅是代码，因为能够编译和运行的代码并不一定实现了所请求的游戏玩法。我们引入了CraftBenchUE，一个评估框架，它在隔离的虚幻引擎环境中运行智能体，在全新的项目中重建其保存的提交，并应用确定性的构建、资产和运行时检查，而无需LLM评判。基于该框架，我们构建了一个包含70个任务的基准测试，涵盖C++源代码、蓝图资产和编辑器脚本。我们在两种编辑器工具配置下评估了七个模型，并在C++任务上使用文件和Shell基线。我们进一步配对指定相同游戏玩法并使用相同运行时测试但要求以C++和蓝图作为交付物的任务。在10个配对任务中，C++完成率在两种工具配置下分别超过蓝图30.0和42.9个百分点。在此配对集中按时提交的蓝图中，通过资产检查的提交中有42.2%和50.0%未能通过显式运行时断言。这些提交满足资产要求，但未能通过所需的游戏玩法测试。我们将随报告发布该框架、任务基准和我们的轨迹发现。
英文摘要
Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay. We introduce CraftBenchUE, an evaluation harness that runs agents in an isolated Unreal Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge. Based on the harness, we built a benchmark consisting of 70 tasks spanning C++ source, Blueprint assets, and editor scripting. We evaluate seven models under two editor-tool configurations, with a file-and-shell baseline on C++ tasks. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C++ and Blueprint as the deliverables. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0 and 42.9 percentage points in the two tool configurations. Among on-time Blueprint submissions in this paired set that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions. These submissions satisfy asset requirements but fail the required gameplay tests. We will release the harness, task benchmark, and our trajectory findings with the report.
