具备3D理解能力的语言-图像模型
Language-Image Models with 3D Understanding
AI总结：
本文提出Cube-LLM，通过构建大规模2D与3D统一问答数据集LV3D进行预训练，在不依赖3D特定架构的情况下显著提升多模态大模型的三维定位与复杂推理能力。
摘要：
多模态大语言模型（MLLMs）已在多种2D视觉与语言任务中展现出卓越能力。我们将MLLMs的感知能力扩展到在三维空间中对图像进行定位与推理。为此，我们首先通过将多个现有2D和3D识别数据集以统一的任务形式——多轮问答——进行组合，构建了一个面向2D和3D的大规模预训练数据集LV3D。接着，我们提出了一种名为Cube-LLM的新型MLLM，并在LV3D上进行预训练。我们证明，纯数据规模化即可带来强大的3D感知能力，而无需针对3D的特定架构设计或训练目标。Cube-LLM表现出与LLM类似的引人注目的特性：（1）Cube-LLM可以应用思维链提示，从2D上下文信息中提升3D理解能力。（2）Cube-LLM能够遵循复杂多样的指令，并适应多种输入和输出格式。（3）Cube-LLM可以通过视觉提示进行引导，例如2D框或来自专家模型的一组候选3D框。我们在室外基准上的实验表明，Cube-LLM显著优于现有基线：在Talk2Car数据集上，3D定位推理的AP-BEV指标提升了21.3个点；在DriveLM数据集上，针对驾驶场景的复杂推理提升了17.7个点。Cube-LLM在通用MLLM基准上也展现了有竞争力的结果，例如在refCOCO上的2D定位平均得分为87.0，以及在VQAv2、GQA、SQA、POPE等视觉问答基准上的复杂推理表现。我们的项目地址为https://janghyuncho.github.io/Cube-LLM。
英文摘要：
Multi-modal large language models (MLLMs) have shown incredible capabilities in a variety of 2D vision and language tasks. We extend MLLMs' perceptual capabilities to ground and reason about images in 3-dimensional space. To that end, we first develop a large-scale pre-training dataset for 2D and 3D called LV3D by combining multiple existing 2D and 3D recognition datasets under a common task formulation: as multi-turn question-answering. Next, we introduce a new MLLM named Cube-LLM and pre-train it on LV3D. We show that pure data scaling makes a strong 3D perception capability without 3D specific architectural design or training objective. Cube-LLM exhibits intriguing properties similar to LLMs: (1) Cube-LLM can apply chain-of-thought prompting to improve 3D understanding from 2D context information. (2) Cube-LLM can follow complex and diverse instructions and adapt to versatile input and output formats. (3) Cube-LLM can be visually prompted such as 2D box or a set of candidate 3D boxes from specialists. Our experiments on outdoor benchmarks demonstrate that Cube-LLM significantly outperforms existing baselines by 21.3 points of AP-BEV on the Talk2Car dataset for 3D grounded reasoning and 17.7 points on the DriveLM dataset for complex reasoning about driving scenarios, respectively. Cube-LLM also shows competitive results in general MLLM benchmarks such as refCOCO for 2D grounding with (87.0) average score, as well as visual question answering benchmarks such as VQAv2, GQA, SQA, POPE, etc. for complex reasoning. Our project is available at https://janghyuncho.github.io/Cube-LLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
