arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.18523cs.CVcs.AI
收藏

MM-Mixing:面向3D理解的多模态混合对齐

MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding

The Chinese University of Hong Kong(香港中文大学);Central South Unversity(中南大学);Zhejiang Lab(之江实验室)

Jiaze Wang, Yi Wang, Ziyu Guo, Renrui Zhang, Donghao Zhou, Guangyong Chen, Anfeng Liu, Pheng-Ann Heng

AI总结:

本研究提出多模态混合对齐框架MM-Mixing,通过两阶段混合训练优化3D编码器,在多个3D理解任务上显著提升性能,且易于集成到现有框架。

摘要:

我们提出了MM-Mixing,一种用于3D理解的多模态混合对齐框架。MM-Mixing将基于混合的方法应用于多模态数据,在增强多样性并改善跨模态对齐效果的同时,保留并优化跨模态连接。我们提出的两阶段训练流程结合了特征级和输入级混合,以优化3D编码器。第一阶段采用带对比学习的特征级混合,将3D特征与其对应模态对齐。第二阶段同时纳入特征级和输入级混合,引入混合点云输入以进一步优化3D特征表示。MM-Mixing增强了模态间关系,提升了泛化能力,确保了特征一致性,同时提供了多样且真实的训练样本。我们证明,MM-Mixing在多种学习场景下显著提升了基线性能,包括零样本3D分类、线性探测3D分类以及跨模态3D形状检索。值得注意的是,我们将ScanObjectNN上的零样本分类准确率从51.3%提升至61.9%,将Objaverse-LVIS上的准确率从46.8%提升至51.4%。我们的研究结果凸显了基于多模态混合的对齐方法在显著推进3D目标识别与理解方面的潜力,同时该方法易于实现并可集成到现有框架中。

英文摘要:

We introduce MM-Mixing, a multi-modal mixing alignment framework for 3D understanding. MM-Mixing applies mixing-based methods to multi-modal data, preserving and optimizing cross-modal connections while enhancing diversity and improving alignment across modalities. Our proposed two-stage training pipeline combines feature-level and input-level mixing to optimize the 3D encoder. The first stage employs feature-level mixing with contrastive learning to align 3D features with their corresponding modalities. The second stage incorporates both feature-level and input-level mixing, introducing mixed point cloud inputs to further refine 3D feature representations. MM-Mixing enhances intermodality relationships, promotes generalization, and ensures feature consistency while providing diverse and realistic training samples. We demonstrate that MM-Mixing significantly improves baseline performance across various learning scenarios, including zero-shot 3D classification, linear probing 3D classification, and cross-modal 3D shape retrieval. Notably, we improved the zero-shot classification accuracy on ScanObjectNN from 51.3% to 61.9%, and on Objaverse-LVIS from 46.8% to 51.4%. Our findings highlight the potential of multi-modal mixing-based alignment to significantly advance 3D object recognition and understanding while remaining straightforward to implement and integrate into existing frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑