MiniGPT-3D：利用2D先验高效对齐3D点云与大型语言模型
MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors
AI总结：
本文提出MiniGPT-3D，利用2D先验和四阶段级联训练策略高效对齐3D点云与LLM，仅用一块RTX 3090训练27小时即在3D分类和描述任务上取得SOTA，并大幅降低训练成本。
摘要：
大型2D视觉-语言模型（2D-LLMs）通过使用简单的投影器将大型语言模型（LLMs）与图像连接起来，已获得广泛关注。受其成功启发，大型3D点云-语言模型（3D-LLMs）也将点云集成到LLMs中。然而，直接将点云与LLM对齐需要昂贵的训练成本，通常在A100上需要数百GPU小时，这阻碍了3D-LLMs的发展。在本文中，我们提出了MiniGPT-3D，一种高效且强大的3D-LLM，仅在一块RTX 3090上训练27小时就取得了多项SOTA结果。具体来说，我们提出利用来自2D-LLMs的2D先验来对齐3D点云与LLMs，这可以利用2D和3D视觉信息之间的相似性。我们引入了一种新颖的四阶段训练策略，以级联方式进行模态对齐，并引入了一个混合查询专家模块，以高效率自适应地聚合特征。此外，我们采用参数高效微调方法LoRA和Norm微调，仅产生47.8M可学习参数，比现有方法最多减少260倍。大量实验表明，MiniGPT-3D在3D物体分类和描述任务上取得了SOTA结果，同时训练成本显著降低。值得注意的是，在具有挑战性的物体描述任务上，与ShapeLLM-13B相比，MiniGPT-3D在GPT-4评估分数上提高了8.12，而后者在8块A800上总共花费160 GPU小时。我们是首个探索高效3D-LLM的工作，为社区提供了新的见解。代码和权重可在https://github.com/TangYuan96/MiniGPT-3D获取。
英文摘要：
Large 2D vision-language models (2D-LLMs) have gained significant attention by bridging Large Language Models (LLMs) with images using a simple projector. Inspired by their success, large 3D point cloud-language models (3D-LLMs) also integrate point clouds into LLMs. However, directly aligning point clouds with LLM requires expensive training costs, typically in hundreds of GPU-hours on A100, which hinders the development of 3D-LLMs. In this paper, we introduce MiniGPT-3D, an efficient and powerful 3D-LLM that achieves multiple SOTA results while training for only 27 hours on one RTX 3090. Specifically, we propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which can leverage the similarity between 2D and 3D visual information. We introduce a novel four-stage training strategy for modality alignment in a cascaded way, and a mixture of query experts module to adaptively aggregate features with high efficiency. Moreover, we utilize parameter-efficient fine-tuning methods LoRA and Norm fine-tuning, resulting in only 47.8M learnable parameters, which is up to 260x fewer than existing methods. Extensive experiments show that MiniGPT-3D achieves SOTA on 3D object classification and captioning tasks, with significantly cheaper training costs. Notably, MiniGPT-3D gains an 8.12 increase on GPT-4 evaluation score for the challenging object captioning task compared to ShapeLLM-13B, while the latter costs 160 total GPU-hours on 8 A800. We are the first to explore the efficient 3D-LLM, offering new insights to the community. Code and weights are available at https://github.com/TangYuan96/MiniGPT-3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
