arXiv 2407.137292025-09-10cs.CL

Baba Is AI：打破规则以超越基准

Baba Is AI: Break the Rules to Beat the Benchmark

MIT（麻省理工学院）；Department of Computer Science, University of Virginia, USA（弗吉尼亚大学计算机科学系）

Nathan Cloos, Meagan Jens, Michelangelo Naim, Yen-Ling Kuo, Ignacio Cases, Andrei Barbu, Christopher J. Cueva

备注：: 8 pages, 8 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文基于游戏《Baba Is You》构建新基准，测试多模态大模型在需操控和组合规则时的泛化能力，发现现有模型表现严重失败。

摘要：

人类通过遵循现有规则和程序来解决问题，同时也通过创造性的飞跃来重新定义这些规则和目标。为了探究这些能力，我们基于游戏《Baba Is You》开发了一个新基准，在该游戏中，智能体需要操控环境中的物体以及规则（规则由写有单词的可移动方块表示），以达到指定目标并赢得游戏。我们测试了三种最先进的多模态大型语言模型（OpenAI GPT-4o、Google Gemini-1.5-Pro 和 Gemini-1.5-Flash），发现当泛化要求必须操控和组合游戏规则时，这些模型表现严重失败。

英文摘要：

Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。