arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06041cs.CV
收藏

从透视描述预测相机姿态用于空间推理

Predicting Camera Pose from Perspective Descriptions for Spatial Reasoning

Xuejun Zhang, Aditi Tiwari, Zhenhailong Wang, Heng Ji

AI总结:

CAMCUE通过姿态感知的多图像框架,从自然语言描述中准确预测相机姿态,提升多视角空间推理效率。

摘要:

多图像空间推理仍然是当前多模态大语言模型(MLLMs)的挑战。虽然单视角感知本质上是二维的,但对多个视角进行推理需要在不同视角之间建立一致的场景理解。特别是,我们研究了视角转换,即模型必须从多视角观测中构建一致的3D理解,并用它从新的语言指定视角进行推理。我们引入了CAMCUE,一种姿态感知的多图像框架,它使用相机姿态作为显式的几何锚点,用于跨视角融合和新视角推理。CAMCUE将每个视角的姿态注入视觉标记中,将自然语言视角描述 grounding 到目标相机姿态,并合成一个姿态条件的想象目标视角以支持回答。为了支持这种设置,我们整理了CAMCUE-DATA,包含27,668个训练和508个测试实例,配对多视角图像和姿态,以及多样化的目标视角描述和视角转换问题。我们还包含人工标注的视角描述在测试集部分以评估对人类语言的泛化能力。CAMCUE将整体准确率提高了9.06%,并且能够从自然语言视角描述中预测目标姿态,其旋转精度在20°以内超过90%,平移精度在0.5误差阈值内。这种直接的 grounding 避免了昂贵的测试时间搜索和匹配,将推理时间从256.6秒减少到每个示例1.45秒,并使在现实世界场景中实现快速、交互式的使用成为可能。

英文摘要:

Multi-image spatial reasoning remains challenging for current multimodal large language models (MLLMs). While single-view perception is inherently 2D, reasoning over multiple views requires building a coherent scene understanding across viewpoints. In particular, we study perspective taking, where a model must build a coherent 3D understanding from multi-view observations and use it to reason from a new, language-specified viewpoint. We introduce CAMCUE, a pose-aware multi-image framework that uses camera pose as an explicit geometric anchor for cross-view fusion and novel-view reasoning. CAMCUE injects per-view pose into visual tokens, grounds natural-language viewpoint descriptions to a target camera pose, and synthesizes a pose-conditioned imagined target view to support answering. To support this setting, we curate CAMCUE-DATA with 27,668 training and 508 test instances pairing multi-view images and poses with diverse target-viewpoint descriptions and perspective-shift questions. We also include human-annotated viewpoint descriptions in the test split to evaluate generalization to human language. CAMCUE improves overall accuracy by 9.06% and predicts target poses from natural-language viewpoint descriptions with over 90% rotation accuracy within 20° and translation accuracy within a 0.5 error threshold. This direct grounding avoids expensive test-time search-and-match, reducing inference time from 256.6s to 1.45s per example and enabling fast, interactive use in real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑