arXiv 2410.002552025-02-20cs.AIcs.CLcs.CV

Robin3D：通过鲁棒指令微调改进3D大型语言模型

Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning

University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Zhejiang University（浙江大学）；University of Central Florida（中佛罗里达大学）；Illinois Institute of Technology（伊利诺伊理工大学）

Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, Yan Yan

备注：: 8 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对3DLLM缺乏鲁棒指令数据的问题，本文提出Robin3D，利用RIG引擎构建百万级对抗与多样化指令数据，并通过关系增强投影器和ID特征绑定提升空间理解与定位能力，在多个3D基准上取得显著性能提升。

摘要：

近期，3D大型语言模型（3DLLM）的进展凸显了其在3D真实世界中构建通用智能体的潜力，但由于缺乏高质量、鲁棒的指令遵循数据，3DLLM仍面临挑战，导致其判别能力和泛化能力受限。本文提出Robin3D，这是一个强大的3DLLM，基于由新型数据引擎Robust Instruction Generation（RIG，鲁棒指令生成）引擎生成的大规模指令遵循数据训练而成。RIG生成两类关键指令数据：1）对抗性指令遵循数据，其特点是混合负样本和正样本，以增强模型的判别性理解；2）多样化指令遵循数据，包含多种指令风格，以增强模型的泛化能力。最终，我们构建了100万条指令遵循数据，包括344K个对抗性样本、508K个多样化样本以及165K个基准训练集样本。为更好地处理这些复杂指令，Robin3D首先引入Relation-Augmented Projector（关系增强投影器）以增强空间理解，随后通过ID-Feature Bonding（ID特征绑定）强化物体指称和定位能力。Robin3D在五个广泛使用的3D多模态学习基准上持续优于以往方法，且无需针对特定任务进行微调。值得注意的是，我们在定位任务（Multi3DRefer）上实现了7.8%的提升，在描述生成任务（Scan2Cap）上实现了6.9%的提升。

英文摘要：

Recent advancements in 3D Large Language Models (3DLLMs) have highlighted their potential in building general-purpose agents in the 3D real world, yet challenges remain due to the lack of high-quality robust instruction-following data, leading to limited discriminative power and generalization of 3DLLMs. In this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale instruction-following data generated by our novel data engine, Robust Instruction Generation (RIG) engine. RIG generates two key instruction data: 1) the Adversarial Instruction-following data, which features mixed negative and positive samples to enhance the model's discriminative understanding. 2) the Diverse Instruction-following data, which contains various instruction styles to enhance model's generalization. As a result, we construct 1 million instruction-following data, consisting of 344K Adversarial samples, 508K Diverse samples, and 165K benchmark training set samples. To better handle these complex instructions, Robin3D first incorporates Relation-Augmented Projector to enhance spatial understanding, and then strengthens the object referring and grounding ability through ID-Feature Bonding. Robin3D consistently outperforms previous methods across five widely-used 3D multimodal learning benchmarks, without the need for task-specific fine-tuning. Notably, we achieve a 7.8\% improvement in the grounding task (Multi3DRefer) and a 6.9\% improvement in the captioning task (Scan2Cap).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。