arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.06555cs.CL
收藏

ING-VP:多模态大语言模型尚无法玩转简单的视觉类游戏

ING-VP: MLLMs cannot Play Easy Vision-based Games Yet

MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Haoran Zhang, Hangyu Guo, Shuyue Guo, Meng Cao, Wenhao Huang, Jiaheng Liu, Ge Zhang

AI总结:

本研究提出首个基于交互式游戏的视觉规划基准ING-VP,用于评估多模态大语言模型的空间想象与多步推理能力,实验发现当前最优模型准确率仅3.37%,为相关能力研究提供了专用评估框架。

摘要:

随着多模态大语言模型(MLLMs)在各类任务中持续展现出愈发强劲的竞争力,研究人员已开发出更复杂、更全面的基准来评估这些前沿模型,这类基准对模型的感知、推理、规划等核心能力提出了新挑战。但现有多模态基准未能针对性评估基于图像空间关系的多步规划能力。为填补这一空白,我们提出了首个基于交互式游戏的视觉规划基准ING-VP,专门用于评估MLLMs的空间想象与多步推理能力。ING-VP包含6款不同的游戏,共300个关卡,每个关卡有6种独特配置,单模型需参与超60000轮交互。该基准框架支持多种对比设置,包括图文输入与纯文本输入对比、单步与多步推理对比、带历史信息与不带历史信息条件对比,可为模型能力分析提供有价值的洞见。我们评估了多款前沿MLLMs,表现最优的模型Claude-3.5 Sonnet的平均准确率仅为3.37%,远低于预期水平。本研究旨在提供一个专用评估框架,以推动MLLMs复杂空间推理与规划能力的发展。相关代码已在https://github.com/Thisisus7/ING-VP.git 公开。

英文摘要:

As multimodal large language models (MLLMs) continue to demonstrate increasingly competitive performance across a broad spectrum of tasks, more intricate and comprehensive benchmarks have been developed to assess these cutting-edge models. These benchmarks introduce new challenges to core capabilities such as perception, reasoning, and planning. However, existing multimodal benchmarks fall short in providing a focused evaluation of multi-step planning based on spatial relationships in images. To bridge this gap, we present ING-VP, the first INteractive Game-based Vision Planning benchmark, specifically designed to evaluate the spatial imagination and multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games, encompassing 300 levels, each with 6 unique configurations. A single model engages in over 60,000 rounds of interaction. The benchmark framework allows for multiple comparison settings, including image-text vs. text-only inputs, single-step vs. multi-step reasoning, and with-history vs. without-history conditions, offering valuable insights into the model's capabilities. We evaluated numerous state-of-the-art MLLMs, with the highest-performing model, Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the anticipated standard. This work aims to provide a specialized evaluation framework to drive advancements in MLLMs' capacity for complex spatial reasoning and planning. The code is publicly available at https://github.com/Thisisus7/ING-VP.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑