推理在3D视觉定位中至关重要
Reasoning Matters for 3D Visual Grounding
AI总结：
本文提出了一种自动合成3D视觉定位数据的管道，并引入了在仅使用1.6%训练数据下表现优于现有方法的Reason3DVG-8B模型，证明了推理在3D视觉定位中的重要性。
摘要：
近年来，具有强大推理能力的大语言模型（LLMs）的发展推动了数学、编码和科学发现等各个领域的研究。与此同时，作为3D理解基本任务的3D视觉定位，仍然具有挑战性，因为最近的3D视觉定位模型的推理能力有限。目前大多数方法结合文本编码器和视觉特征编码器，生成跨模态融合特征并预测指代对象。这些模型通常需要在大量3D标注数据上进行监督训练。另一方面，近期研究也聚焦于通过合成数据训练更强的3D视觉定位LLM，然而性能提升仍然有限，且与数据收集成本不成正比。在本工作中，我们提出了一条3D视觉定位数据管道，能够自动合成3D视觉定位数据并生成相应的推理过程。此外，我们利用生成的数据对LLM进行微调，并引入Reason3DVG-8B，一个强大的3D视觉定位LLM，仅使用其训练数据的1.6%，就超过了先前基于LLM的方法3D-GRAND，证明了我们数据的有效性以及推理在3D视觉定位中的重要性。
英文摘要：
The recent development of Large Language Models (LLMs) with strong reasoning ability has driven research in various domains such as mathematics, coding, and scientific discovery. Meanwhile, 3D visual grounding, as a fundamental task in 3D understanding, still remains challenging due to the limited reasoning ability of recent 3D visual grounding models. Most of the current methods incorporate a text encoder and visual feature encoder to generate cross-modal fuse features and predict the referring object. These models often require supervised training on extensive 3D annotation data. On the other hand, recent research also focus on scaling synthetic data to train stronger 3D visual grounding LLM, however, the performance gain remains limited and non-proportional to the data collection cost. In this work, we propose a 3D visual grounding data pipeline, which is capable of automatically synthesizing 3D visual grounding data along with corresponding reasoning process. Additionally, we leverage the generated data for LLM fine-tuning and introduce Reason3DVG-8B, a strong 3D visual grounding LLM that outperforms previous LLM-based method 3D-GRAND using only 1.6% of their training data, demonstrating the effectiveness of our data and the importance of reasoning in 3D visual grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
