视感知-aware的多模态空间推理从单目图像
Perception-Aware Multimodal Spatial Reasoning from Monocular Images
AI总结：
本文提出了一种感知-aware的多模态推理框架，通过统一标记空间联合处理视觉证据和文本推理，提升单目图像空间推理的鲁棒性。
摘要：
从单目图像进行空间推理对于自动驾驶至关重要，然而当前的视觉-语言模型（VLMs）仍然在细粒度几何感知方面存在困难，尤其是在大规模变化和模糊物体外观的情况下。我们提出了一种简单而有效的感知-aware多模态推理框架，使VLMs具备显式的以对象为中心的定位能力。与其依赖文本边界框输出不同，每个被引用的对象都使用其空间范围内所有视觉参考标记（VRTs）进行表示，使视觉证据和文本推理能够在统一的标记空间中联合处理。为进一步加强跨模态交互，我们构建了一个多模态链式思考（MM-CoT）数据集，该数据集注入了对齐的视觉和文本推理信号。引入了一种确定性的排序策略，使对本质上无序的VRT集合的监督能够完全与VLM的自回归下一个标记预测兼容。仅通过标准监督微调，我们的方法在SURDS基准上实现了显著的改进，在单对象和多对象任务中均优于先前的方法，包括那些使用基于强化学习的后训练方法的。这些结果表明，精确的感知和多模态推理是相互促进的，共同构成了在具有挑战性的单目驾驶场景中实现稳健空间理解的关键。
英文摘要：
Spatial reasoning from monocular images is essential for autonomous driving, yet current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception, particularly under large scale variation and ambiguous object appearance. We propose a simple yet effective perception-aware multimodal reasoning framework that equips VLMs with explicit object-centric grounding ability. Instead of relying on textual bounding-box outputs, each referred object is represented using all Visual Reference Tokens (VRTs) within its spatial extent, enabling visual evidence and textual reasoning to be processed jointly in a unified token space. To further strengthen cross-modal interaction, we construct a Multimodal Chain-of-Thought (MM-CoT) dataset that injects aligned visual and textual reasoning signals. A deterministic ordering strategy is introduced to make supervision over inherently unordered VRT sets fully compatible with the VLM's autoregressive next-token prediction. With only standard supervised fine-tuning, our method achieves substantial improvements on the SURDS benchmark, outperforming previous approaches - including those using RL-based post-training - by a large margin across both single-object and multi-object tasks. These results demonstrate that accurate perception and multimodal reasoning are mutually reinforcing, and together form the key to robust spatial understanding in challenging monocular driving scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
