ActiView：评估多模态大语言模型的主动感知能力
ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models
AI总结：
本文提出ActiView基准，通过限制感知视野并要求模型主动缩放或移动视野来回答问题，系统评估了30个多模态大语言模型的主动感知能力，揭示了现有模型在该能力上的显著不足。
摘要：
主动感知是人类的一项关键能力，它涉及基于对环境的当前理解设定目标，并执行动作以实现该目标。尽管在多模态大语言模型（MLLMs）的评估方面已有大量工作，但主动感知在很大程度上被忽视了。为填补这一空白，我们提出了一个名为ActiView的新型基准，用于评估MLLMs的主动感知能力。我们聚焦于一种特殊形式的视觉问答（VQA），它简化并量化了评估过程，同时对现有MLLMs仍具有挑战性。同时，我们还讨论了模型的中间推理行为。给定一张图像，我们限制模型的感知视野，要求其基于推理主动缩放或移动感知视野，以成功回答问题。我们对30个模型进行了广泛评估，包括专有模型和开源模型，并观察到受限的感知视野在实现主动感知方面发挥着重要作用。结果表明，MLLMs的主动感知能力存在显著差距，表明这一领域值得更多关注。我们希望ActiView能够帮助开发方法，使MLLMs以更自然和整体的方式理解多模态输入。
英文摘要：
Active perception, a crucial human capability, involves setting a goal based on the current understanding of the environment and performing actions to achieve that goal. Despite significant efforts in evaluating Multimodal Large Language Models (MLLMs), active perception has been largely overlooked. To address this gap, we propose a novel benchmark named ActiView to evaluate active perception in MLLMs. We focus on a specialized form of Visual Question Answering (VQA) that eases and quantifies the evaluation yet challenging for existing MLLMs. Meanwhile, intermediate reasoning behaviors of models are also discussed. Given an image, we restrict the perceptual field of a model, requiring it to actively zoom or shift its perceptual field based on reasoning to answer the question successfully. We conduct extensive evaluation over 30 models, including proprietary and open-source models, and observe that restricted perceptual fields play a significant role in enabling active perception. Results reveal a significant gap in the active perception capability of MLLMs, indicating that this area deserves more attention. We hope that ActiView could help develop methods for MLLMs to understand multimodal inputs in more natural and holistic ways.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
