发表机构
The Chinese University of Hong Kong; Nankai University; Wuhan University（香港中文大学; 南开大学; 武汉大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
Spec2Game基准通过15个游戏家族和150个任务实例评估LLM从自然语言规格生成完整Pygame项目的能力，发现高可执行性不保证规格忠实实现，规则与终止逻辑建模仍是主要挑战。
AI 中文摘要
生成可执行程序并不一定意味着它正确实现了自然语言中指定的行为需求。为了评估大型语言模型将详细规格实现为完整交互式程序的能力，我们引入了Spec2Game，这是一个要求模型从详细的自然语言游戏规格中生成完整Pygame项目的基准。Spec2Game包含15个游戏家族和150个任务实例，每个家族有一个规范任务和九个受控规则变体，涵盖三个实现复杂度级别。利用源代码、运行时和视觉证据，我们从四个维度评估生成的项目——可执行性、规格实现、代码质量和用户面向质量。在14个大型语言模型和3,330个生成项目中，我们发现高可执行性并不意味著忠实的规格实现。组件级分析进一步表明，模型在游戏元素建模方面的表现明显优于规则与机制建模或目标与终止建模，这表明忠实地实现游戏规则和终止逻辑仍然是一个重大挑战。
英文摘要
Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language. To evaluate large language models' ability to realize detailed specifications as complete interactive programs, we introduce Spec2Game, a benchmark that requires models to generate complete Pygame projects from detailed natural-language game specifications. Spec2Game comprises 15 game families and 150 task instances, with one canonical task and nine controlled rule variants per family, spanning three levels of implementation complexity. Using source-code, runtime, and visual evidence, we evaluate generated projects along four dimensions---Executability, Specification Realization, Code Quality, and User-Facing Quality. Across 14 LLMs and 3,330 generated projects, we find that high executability does not imply faithful specification realization. Component-level analysis further shows that models perform substantially better on Game Element Modeling than on Rule and Mechanism Modeling or Goal and Termination Modeling, indicating that faithfully implementing game rules and termination logic remains a major challenge.
Comments37 pages, 11 figures
