From Code to Play: Benchmarking Program Search for Games Using Large Language Models
Institute of Applied AI, Stuttgart Media University(应用人工智能研究所,斯图加特媒体大学);School of Electronic Engineering and Computer Science, Queen Mary University of London(电子工程与计算机科学学院,伦敦女王大学);Institute for Information Processing, Leibniz University Hannover(信息处理研究所,汉诺威莱布尼茨大学);Creative Assembly(创意装配);Chair of Learning Technical Systems, Ruhr-University Bochum(学习技术系统教授职位,博德鲁姆鲁尔大学)
Manuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas
备注:
Submitted to Transactions on Games Special Issue on Large Language Models and Games, standardised LLMs used and run more experiments
大语言模型(LLMs)在生成程序代码方面展现出令人印象深刻的能力,为将程序合成应用于游戏开启了激动人心的机遇。在本研究中,我们探索了 LLMs 直接为广泛的游戏应用合成可用代码的潜力,重点关注 Python 和 Java 两种编程语言。我们使用了一种进化爬山算法,其中初始程序的变异和种子由 LLMs 控制。对于 Python,该框架涵盖了各种与游戏相关的任务,包括 5 个 Atari 游戏的微型版本、10 个 Baba is You 关卡、一个受 Asteroids 启发的环境以及一个迷宫生成任务。对于 Java,该框架包含来自 TAG 桌面游戏框架的 12 款游戏。在 29 个任务中,我们评估了 12 个用于 Python 的语言模型和 8 个用于 Java 的语言模型。我们的研究结果表明,LLMs 的性能更多取决于任务而非模型规模。虽然较大的模型能生成更多可执行程序,但这些程序并不总能带来更高质量的解决方案,且成本要高得多。没有哪个模型具有绝对优势,尽管在特定任务上某个模型可能表现更好。在问题上尝试多个模型并利用其中的最佳结果,比仅使用单一模型更可靠。
英文摘要:
Large language models (LLMs) have shown impressive capabilities in generating program code, opening exciting opportunities for applying program synthesis to games. In this work, we explore the potential of LLMs to directly synthesize usable code for a wide range of gaming applications, focusing on two programming languages, Python and Java. We use an evolutionary hill-climbing algorithm, where the mutations and seeds of the initial programs are controlled by LLMs. For Python, the framework covers various game-related tasks, including five miniature versions of Atari games, ten levels of Baba is You, an environment inspired by Asteroids, and a maze generation task. For Java, the framework contains 12 games from the TAG tabletop games framework. Across 29 tasks, we evaluated 12 language models for Python and 8 for Java. Our findings suggest that the performance of LLMs depends more on the task than on model size. While larger models generate more executable programs, these do not always result in higher-quality solutions but are much more expensive. No model has a clear advantage, although on any specific task, one model may be better. Trying many models on a problem and using the best results across them is more reliable than using just one.
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
University College London(伦敦大学学院);IDEAS NCBR;University of Oxford(牛津大学);New York University(纽约大学);Anthropic;University of Warsaw(华沙大学);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所)
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
MBZUAI(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);Hong Kong Baptist University(香港浸会大学);University of Oxford(牛津大学);New York University, Abu Dhabi(纽约大学阿布扎比分校)
InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Key Laboratory of Cyberspace Security, Ministry of Education(教育部网络空间安全重点实验室);Hubei Key Laboratory of Distributed System Security(湖北省分布式系统安全重点实验室)
CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas
Carnegie Mellon University;Foundations of Cooperative AI Lab (FOCAL);Jinesis Lab, University of Toronto \& Vector Institute;ETH Z\" u rich;Max Planck Institute for Intelligent Systems, T\" u bingen, Germany
GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theory
ETH Zürich(苏黎世联邦理工学院);Berea College(贝雷学院);University of Toronto(多伦多大学);Vector Institute(向量研究所);Max Planck Institute for Intelligent Systems, Tübingen, Germany(图宾根德国智能系统马克斯·普朗克研究所)
The Automated but Risky Game: Modeling and Benchmarking Agent-to-Agent Negotiations and Transactions in Consumer Markets
Stanford University(斯坦福大学);University of Toronto(多伦多大学);Google DeepMind(谷歌DeepMind);University of Southern California(南加州大学);Massachusetts Institute of Technology(麻省理工学院)
On game psychology: an experiment on the chess board/screen, should you always "do your best", and why the programs with prescribed weaknesses cannot be our good friends?
Kinneret College on the Sea of Galilee(加利利海金内雷特学院);Braude Academic College(布劳德学术学院);Ben Gurion University of the Negev(内盖夫本古里安大学)