arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.15950cs.AI
收藏

Atari-GPT:评估多模态大型语言模型作为Atari游戏底层策略的基准

Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games

DEVCOM Army Research Laboratory(美国陆军作战能力发展司令部陆军研究实验室);Army Educational Outreach Program(陆军教育拓展计划)

Nicholas R. Waytowich, Devin White, MD Sunbeam, Vinicius G. Goecks

AI总结:

本文提出Atari-GPT基准,评估多模态大型语言模型作为Atari游戏底层策略的能力,发现其尚未能成为零样本底层策略,部分原因是视觉与空间推理不足,为相关研究提供测试基准。

摘要:

大型语言模型(LLMs)的最新进展已将其能力从传统文本任务扩展至多模态领域,整合视觉、听觉和文本数据。尽管多模态LLMs在机器人和游戏等高阶规划领域已被广泛探索,但它们作为底层控制器的潜力仍未得到充分开发。本文提出一种新基准,旨在测试多模态LLMs作为Atari游戏底层策略的涌现能力。与需针对每个新环境训练并指定奖励函数的传统强化学习(RL)方法不同,这些LLMs利用已有多模态知识直接与游戏环境交互。我们的研究评估了多个多模态LLMs与传统RL智能体、人类玩家及随机智能体的性能,重点关注它们理解复杂视觉场景、交互并制定策略响应的能力。结果显示,这些多模态LLMs尚未能成为零样本底层策略,部分原因在于其视觉与空间推理能力不足。更多结果和视频可在项目网页查看:https://dev1nw.github.io/atari-gpt/

英文摘要:

Recent advancements in large language models (LLMs) have expanded their capabilities beyond traditional text-based tasks to multimodal domains, integrating visual, auditory, and textual data. While multimodal LLMs have been extensively explored for high-level planning in domains like robotics and games, their potential as low-level controllers remains largely untapped. In this paper, we introduce a novel benchmark aimed at testing the emergent capabilities of multimodal LLMs as low-level policies in Atari games. Unlike traditional reinforcement learning (RL) methods that require training for each new environment and reward function specification, these LLMs utilize pre-existing multimodal knowledge to directly engage with game environments. Our study assesses the performances of multiple multimodal LLMs against traditional RL agents, human players, and random agents, focusing on their ability to understand and interact with complex visual scenes and formulate strategic responses. Our results show that these multimodal LLMs are not yet capable of being zero-shot low-level policies. Furthermore, we see that this is, in part, due to their visual and spatial reasoning. Additional results and videos are available on our project webpage: https://dev1nw.github.io/atari-gpt/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑