arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.07989cs.CVcs.AIcs.CLcs.LGcs.SDeess.AS
收藏

Any2Point:赋能任意模态大模型以实现高效3D理解

Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding

Shanghai AI Lab(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Northwestern Polytechnical University(西北工业大学)

Yiwen Tang, Ray Zhang, Jiaming Liu, Zoey Guo, Dong Wang, Zhigang Wang, Bin Zhao, Shanghang Zhang, Peng Gao, Hongsheng Li, Xuelong Li

AI总结:

Any2Point提出一种参数高效方法,通过3D到任意模态的虚拟投影和任意到3D引导适配器,赋能视觉、语言、音频等任意模态大模型进行高效3D理解,避免几何损失并降低计算成本。

摘要:

大型基础模型近来已成为备受关注的焦点,在广泛场景中取得了优越性能。由于3D数据稀缺,许多工作致力于将预训练的视觉Transformer适配到3D领域。然而,这种2D到3D的方法仍然存在局限,原因在于可能丢失空间几何信息且计算成本高昂。更重要的是,这些框架主要针对2D模型设计,缺乏通用的任意模态到3D的范式。本文提出Any2Point,一种参数高效的方法,用于赋能任意模态大模型(视觉、语言、音频)进行3D理解。给定一个来自任意源模态的冻结Transformer,我们提出一种3D到任意(1D或2D)虚拟投影策略,将输入的3D点关联到源模态内的原始1D或2D位置。该机制使我们能够为每个3D token分配一个与预训练模型配对的位置编码,从而避免了真实投影造成的3D几何损失,并更好地借助1D/2D位置先验激励Transformer进行3D学习。然后,在每个Transformer块内,我们插入一个任意到3D的引导适配器模块,用于参数高效微调。该适配器融合来自源模态的先验空间知识,以引导3D token的局部特征聚合,促使任意模态Transformer完成语义适配。我们进行了大量实验,以展示所提方法的有效性和高效性。代码和模型已在 https://github.com/Ivan-Tang-3D/Any2Point 发布。

英文摘要:

Large foundation models have recently emerged as a prominent focus of interest, attaining superior performance in widespread scenarios. Due to the scarcity of 3D data, many efforts have been made to adapt pre-trained transformers from vision to 3D domains. However, such 2D-to-3D approaches are still limited, due to the potential loss of spatial geometries and high computation cost. More importantly, their frameworks are mainly designed for 2D models, lacking a general any-to-3D paradigm. In this paper, we introduce Any2Point, a parameter-efficient method to empower any-modality large models (vision, language, audio) for 3D understanding. Given a frozen transformer from any source modality, we propose a 3D-to-any (1D or 2D) virtual projection strategy that correlates the input 3D points to the original 1D or 2D positions within the source modality. This mechanism enables us to assign each 3D token with a positional encoding paired with the pre-trained model, which avoids 3D geometry loss caused by the true projection and better motivates the transformer for 3D learning with 1D/2D positional priors. Then, within each transformer block, we insert an any-to-3D guided adapter module for parameter-efficient fine-tuning. The adapter incorporates prior spatial knowledge from the source modality to guide the local feature aggregation of 3D tokens, compelling the semantic adaption of any-modality transformers. We conduct extensive experiments to showcase the effectiveness and efficiency of our method. Code and models are released at https://github.com/Ivan-Tang-3D/Any2Point.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑