视觉语言模型中的视角感知推理：基于心理意象模拟
Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation
AI总结：
本文提出APC框架，利用视觉基础模型构建场景抽象，通过心理意象模拟实现视角转换，显著提升VLM的视角感知推理能力，优于现有微调和合成方法。
摘要：
我们提出了一种通过心理意象模拟在视觉语言模型（VLMs）中进行视角感知推理的框架。视角采择（perspective-taking），即从替代视角感知环境或情境的能力，是衡量人类级视觉理解的关键基准，对于环境交互和与自主代理的协作至关重要。尽管VLM在空间推理方面取得了进展，但近期研究表明，现代VLM显著缺乏视角感知推理能力，并表现出对自我中心解释的强烈偏见。为弥合VLM与人类感知之间的差距，我们聚焦于心理意象的作用，人类通过抽象表征来感知世界，从而促进视角转换。受此启发，我们提出了一种名为抽象视角转换（APC）的视角感知推理框架，该框架有效利用视觉基础模型（如目标检测、分割和方向估计）来构建场景抽象并实现视角变换。我们在合成和真实图像基准上的实验，与多种VLM进行比较，证明了我们的框架在视角感知推理方面取得了显著改进，且进一步优于微调的空间推理模型和基于新视图合成的方法。
英文摘要：
We present a framework for perspective-aware reasoning in vision-language models (VLMs) through mental imagery simulation. Perspective-taking, the ability to perceive an environment or situation from an alternative viewpoint, is a key benchmark for human-level visual understanding, essential for environmental interaction and collaboration with autonomous agents. Despite advancements in spatial reasoning within VLMs, recent research has shown that modern VLMs significantly lack perspective-aware reasoning capabilities and exhibit a strong bias toward egocentric interpretations. To bridge the gap between VLMs and human perception, we focus on the role of mental imagery, where humans perceive the world through abstracted representations that facilitate perspective shifts. Motivated by this, we propose a framework for perspective-aware reasoning, named Abstract Perspective Change (APC), that effectively leverages vision foundation models, such as object detection, segmentation, and orientation estimation, to construct scene abstractions and enable perspective transformations. Our experiments on synthetic and real-image benchmarks, compared with various VLMs, demonstrate significant improvements in perspective-aware reasoning with our framework, further outperforming fine-tuned spatial reasoning models and novel-view-synthesis-based approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
