从对应到动作：多模态大语言模型中类人多图像空间推理
From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models
AI总结：
提出HATCH框架，通过补丁级空间对齐和动作-答案推理两个目标，提升多模态大模型在多图像空间推理中的性能，在三个基准上超越同规模基线。
摘要：
尽管多模态大语言模型（MLLMs）在单图像空间推理方面取得了实质性进展，但多图像空间推理（需要整合来自多个视角的信息）仍然具有挑战性。认知研究表明，人类通过两种机制处理此类任务：跨视图对应（识别不同视图中对应于相同物理位置的区域）和逐步视角变换（顺序组合相对视角变化）。然而，现有研究仅部分且通常隐式地整合这些机制，没有对两者进行显式监督。我们提出了用于跨视图对应和视角变化的类人感知训练（HATCH），这是一个具有两个互补目标的训练框架：（1）补丁级空间对齐，鼓励补丁表示在空间对应区域跨视图对齐；（2）动作-答案推理，要求模型在预测最终答案之前生成显式的视角转换动作。在三个基准上的实验表明，HATCH以明显优势持续优于同规模基线，并与更大的模型相比取得了有竞争力的结果，同时保持了单图像推理能力。
英文摘要：
While multimodal large language models (MLLMs) have made substantial progress in single-image spatial reasoning, multi-image spatial reasoning, which requires integration of information from multiple viewpoints, remains challenging. Cognitive studies suggest that humans address such tasks through two mechanisms: cross-view correspondence, which identifies regions across different views that correspond to the same physical locations, and stepwise viewpoint transformation, which composes relative viewpoint changes sequentially. However, existing studies incorporate these mechanisms only partially and often implicitly, without explicit supervision for both. We propose Human-Aware Training for Cross-view correspondence and viewpoint cHange (HATCH), a training framework with two complementary objectives: (1) Patch-Level Spatial Alignment, which encourages patch representations to align across views for spatially corresponding regions, and (2) Action-then-Answer Reasoning, which requires the model to generate explicit viewpoint transition actions before predicting the final answer. Experiments on three benchmarks demonstrate that HATCH consistently outperforms baselines of comparable size by a clear margin and achieves competitive results against much larger models, while preserving single-image reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
