arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04057cs.AI
收藏

从代码到游戏:基于大语言模型的游戏程序搜索基准测试

From Code to Play: Benchmarking Program Search for Games Using Large Language Models

Institute of Applied AI, Stuttgart Media University(应用人工智能研究所,斯图加特媒体大学);School of Electronic Engineering and Computer Science, Queen Mary University of London(电子工程与计算机科学学院,伦敦女王大学);Institute for Information Processing, Leibniz University Hannover(信息处理研究所,汉诺威莱布尼茨大学);Creative Assembly(创意装配);Chair of Learning Technical Systems, Ruhr-University Bochum(学习技术系统教授职位,博德鲁姆鲁尔大学)

Manuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas

AI总结:

本研究探索大语言模型结合进化爬山算法为Python和Java游戏合成代码的潜力,在29个任务上评估20个模型,发现性能取决于任务而非模型规模,多模型择优比单模型更可靠。

摘要:

大语言模型(LLMs)在生成程序代码方面展现出令人印象深刻的能力,为将程序合成应用于游戏开启了激动人心的机遇。在本研究中,我们探索了 LLMs 直接为广泛的游戏应用合成可用代码的潜力,重点关注 Python 和 Java 两种编程语言。我们使用了一种进化爬山算法,其中初始程序的变异和种子由 LLMs 控制。对于 Python,该框架涵盖了各种与游戏相关的任务,包括 5 个 Atari 游戏的微型版本、10 个 Baba is You 关卡、一个受 Asteroids 启发的环境以及一个迷宫生成任务。对于 Java,该框架包含来自 TAG 桌面游戏框架的 12 款游戏。在 29 个任务中,我们评估了 12 个用于 Python 的语言模型和 8 个用于 Java 的语言模型。我们的研究结果表明,LLMs 的性能更多取决于任务而非模型规模。虽然较大的模型能生成更多可执行程序,但这些程序并不总能带来更高质量的解决方案,且成本要高得多。没有哪个模型具有绝对优势,尽管在特定任务上某个模型可能表现更好。在问题上尝试多个模型并利用其中的最佳结果,比仅使用单一模型更可靠。

英文摘要:

Large language models (LLMs) have shown impressive capabilities in generating program code, opening exciting opportunities for applying program synthesis to games. In this work, we explore the potential of LLMs to directly synthesize usable code for a wide range of gaming applications, focusing on two programming languages, Python and Java. We use an evolutionary hill-climbing algorithm, where the mutations and seeds of the initial programs are controlled by LLMs. For Python, the framework covers various game-related tasks, including five miniature versions of Atari games, ten levels of Baba is You, an environment inspired by Asteroids, and a maze generation task. For Java, the framework contains 12 games from the TAG tabletop games framework. Across 29 tasks, we evaluated 12 language models for Python and 8 for Java. Our findings suggest that the performance of LLMs depends more on the task than on model size. While larger models generate more executable programs, these do not always result in higher-quality solutions but are much more expensive. No model has a clear advantage, although on any specific task, one model may be better. Trying many models on a problem and using the best results across them is more reliable than using just one.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑