arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.23945cs.CVcs.AIcs.MM
收藏

PointCoT: 一种用于显式3D几何推理的多模态基准

PointCoT: A Multi-modal Benchmark for Explicit 3D Geometric Reasoning

Xi'an Jiaotong University(西安交通大学);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Institute of Automation, CASIA(中国科学院自动化研究所);Taiyuan University of Technology(太原理工大学);Shanghai AI Laboratory(上海人工智能实验室)
Dongxu Zhang, Yiding Sun, Pengcheng Li, Yumou Liu, Hongqiang Lin, Haoran Xu, Xiaoxuan Mu, Liang Lin, Wenbiao Yan, Ning Yang, Chaowei Fang, Juanjuan Zhao, Jihua … 展开作者

Dongxu Zhang, Yiding Sun, Pengcheng Li, Yumou Liu, Hongqiang Lin, Haoran Xu, Xiaoxuan Mu, Liang Lin, Wenbiao Yan, Ning Yang, Chaowei Fang, Juanjuan Zhao, Jihua Zhu, Conghui He, Cheng Tan

AI总结:

PointCoT通过显式链式推理提升3D几何推理能力,提出多模态基准和双流架构,实现对3D点云的高精度理解与推理。

摘要:

尽管多模态大语言模型(MLLMs)在2D场景中表现出色,但将其感知智能扩展到3D点云理解仍是一个重大挑战。当前方法主要集中在将3D特征与预训练模型对齐。然而,它们通常将几何推理视为隐式映射过程。这些方法绕过了中间的逻辑步骤,因此导致几何幻觉。它们会自信地生成合理但不基于精确结构细节的响应。为了填补这一差距,我们提出了PointCoT,一种新的框架,使MLLMs能够为3D数据提供显式的链式推理(CoT)。我们倡导一种“观察、思考、然后回答”的范式。在这种方法中,模型被监督生成基于几何的推理过程,然后再预测最终答案。为了促进这一点,我们构建了Point-Reason-Instruct,一个包含约86,000个指令训练样本的大型基准,具有分层的CoT注释。通过利用双流多模态架构,我们的方法结合了语义外观与几何真实性。广泛的实验表明,PointCoT在复杂推理任务上实现了最先进的性能。

英文摘要:

While Multimodal Large Language Models (MLLMs) demonstrate proficiency in 2D scenes, extending their perceptual intelligence to 3D point cloud understanding remains a significant challenge. Current approaches focus primarily on aligning 3D features with pre-trained models. However, they typically treat geometric reasoning as an implicit mapping process. These methods bypass intermediate logical steps and consequently suffer from geometric hallucinations. They confidently generate plausible responses that fail to ground in precise structural details. To bridge this gap, we present PointCoT, a novel framework that empowers MLLMs with explicit Chain-of-Thought (CoT) reasoning for 3D data. We advocate for a \textit{Look, Think, then Answer} paradigm. In this approach, the model is supervised to generate geometry-grounded rationales before predicting final answers. To facilitate this, we construct Point-Reason-Instruct, a large-scale benchmark comprising $\sim$86k instruction-tuning samples with hierarchical CoT annotations. By leveraging a dual-stream multi-modal architecture, our method synergizes semantic appearance with geometric truth. Extensive experiments demonstrate that PointCoT achieves state-of-the-art performance on complex reasoning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑