从另一视角观察：评估多模态大语言模型中的多视角理解能力
Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs
AI总结：
针对MLLMs在多视角几何一致性和跨视角对应上的不足，提出包含2100多个QA对的All-Angles Bench基准，通过六项任务全面评估27个模型，发现当前模型在部分遮挡对应和相机位姿估计上远不及人类水平。
摘要：
多视角理解，即协调不同视点的视觉信息以实现有效导航、操作和3D场景理解的能力，是多模态大语言模型（MLLMs）作为具身智能体所面临的一项根本性挑战。尽管近期的MLLMs在高级推理和规划方面取得了令人瞩目的进展，但在面对多视角几何一致性和跨视角对应关系时，它们往往表现不佳。为了全面评估MLLMs在多视角场景推理中的挑战，我们提出了All-Angles Bench，这是一个包含90个多样化真实世界场景、超过2100个人工精心标注的多视角问答对的基准。我们的六项任务（计数、属性识别、相对距离、相对方向、物体操作和相机位姿估计）专门测试模型的几何对应关系以及在不同视角间一致对齐信息的能力。我们针对27个具有代表性的MLLMs（包括Gemini-2.0-Flash、Claude-3.7-Sonnet和GPT-4o）进行了广泛实验，并与人类评估者进行了基准对比，结果揭示了巨大的性能差距，表明当前MLLMs仍远未达到人类水平的熟练度。通过深入分析，我们发现MLLMs在以下两个方面表现尤为不佳：(1)部分遮挡视角下的跨视角对应关系，以及(2)建立粗略的相机位姿。这些发现凸显了引入特定领域微调或嵌入更强多视角感知模块的必要性。我们相信，All-Angles Bench提供了宝贵的见解，有助于弥合MLLMs与人类水平多视角理解之间的差距。该项目和基准已公开发布于https://danielchyeh.github.io/All-Angles-Bench/。
英文摘要：
Multi-view understanding, the ability to reconcile visual information across diverse viewpoints for effective navigation, manipulation, and 3D scene comprehension, is a fundamental challenge in Multi-Modal Large Language Models (MLLMs) to be used as embodied agents. While recent MLLMs have shown impressive advances in high-level reasoning and planning, they frequently fall short when confronted with multi-view geometric consistency and cross-view correspondence. To comprehensively evaluate the challenges of MLLMs in multi-view scene reasoning, we propose All-Angles Bench, a benchmark of over 2,100 human carefully annotated multi-view question-answer pairs across 90 diverse real-world scenes. Our six tasks (counting, attribute identification, relative distance, relative direction, object manipulation, and camera pose estimation) specifically test model's geometric correspondence and the capacity to align information consistently across views. Our extensive experiments, benchmark on 27 representative MLLMs including Gemini-2.0-Flash, Claude-3.7-Sonnet, and GPT-4o against human evaluators reveals a substantial performance gap, indicating that current MLLMs remain far from human-level proficiency. Through in-depth analysis, we show that MLLMs are particularly underperforming under two aspects: (1) cross-view correspondence for partially occluded views and (2) establishing the coarse camera poses. These findings highlight the necessity of domain-specific refinements or modules that embed stronger multi-view awareness. We believe that our All-Angles Bench offers valuable insights and contribute to bridging the gap between MLLMs and human-level multi-view understanding. The project and benchmark are publicly available at https://danielchyeh.github.io/All-Angles-Bench/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
