We introduce MM-Mixing, a multi-modal mixing alignment framework for 3D understanding. MM-Mixing applies mixing-based methods to multi-modal data, preserving and optimizing cross-modal connections while enhancing diversity and improving alignment across modalities. Our proposed two-stage training pipeline combines feature-level and input-level mixing to optimize the 3D encoder. The first stage employs feature-level mixing with contrastive learning to align 3D features with their corresponding modalities. The second stage incorporates both feature-level and input-level mixing, introducing mixed point cloud inputs to further refine 3D feature representations. MM-Mixing enhances intermodality relationships, promotes generalization, and ensures feature consistency while providing diverse and realistic training samples. We demonstrate that MM-Mixing significantly improves baseline performance across various learning scenarios, including zero-shot 3D classification, linear probing 3D classification, and cross-modal 3D shape retrieval. Notably, we improved the zero-shot classification accuracy on ScanObjectNN from 51.3% to 61.9%, and on Objaverse-LVIS from 46.8% to 51.4%. Our findings highlight the potential of multi-modal mixing-based alignment to significantly advance 3D object recognition and understanding while remaining straightforward to implement and integrate into existing frameworks.
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Peking University(北京大学);Tencent Data Platform(腾讯数据平台);Pengcheng Lab(鹏城实验室);National University of Singapore(新加坡国立大学);Nari Technology Development Limited Company(南瑞集团有限公司)
针对视频-语言预训练难以扩展到 N 模态的问题,本文提出 LanguageBind,以语言为纽带通过冻结语言编码器和对比学习对齐多模态,并构建 VIDAL-10M 数据集,在 15 个基准上验证有效性。
MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention
The University of Sydney(悉尼大学);School of Computer Science, The University of Sydney(悉尼大学计算机科学学院);National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology(集成空天地海大数据应用技术国家工程实验室);School of Computer Science and Engineering, Northwestern Polytechnical University(西北工业大学计算机科学与工程学院);University of Maryland(马里兰大学);Ningbo Institute of Northwestern Polytechnical University(西北工业大学宁波学院)
MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS研究室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);GIPSA-lab, Univ. Grenoble Alpes, CNRS, Grenoble INP(格勒诺布尔大学GIPSA实验室,法国国家科学研究中心,格勒诺布尔国立综合理工学院)
InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Shanghai Jiao Tong University(上海交通大学);Hong Kong University of Science and Technology(香港科技大学)
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment
Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Sun Yat-sen University(中山大学)
AnANet: Modeling Association and Alignment for Cross-modal Correlation Classification
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Wenge Technology Co.,Ltd(北京文阁科技有限公司)
AlignFreeNet: Is Cross-Modal Pre-Alignment Necessary? An End-to-End Alignment-Free Lightweight Network for Visible-Infrared Object Detection
School of Computer Science, Jiangsu University of Technology(江苏科技大学计算机科学学院);School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);School of Science and Technology, Hong Kong Metropolitan University(香港都会大学科技学院);School of Data Science, Lingnan University(岭南大学数据科学学院);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)
Robust Incomplete-Modality Alignment for Ophthalmic Disease Grading and Diagnosis via Labeled Optimal Transport
University of Exeter(埃克塞特大学);University of Liverpool(利物浦大学);Chinese Academy of Sciences(中国科学院);The University of Hong Kong(香港大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Dalian University of Technology(大连理工大学);A*STAR
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment
Zhejiang University(浙江大学);The University of Manchester(曼彻斯特大学);University of Oxford(牛津大学);The University of Edinburgh(爱丁堡大学);Tsinghua University(清华大学);Donghai laboratory(东海实验室)
Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment
Peking University(北京大学);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Computer Science, Peking University(北京大学计算机学院)
FUSE: Frequency-domain Unification and Spectral Energy Alignment for Multi-modal Object Re-Identification
School of Cyber Science and Engineering, Xi'an Jiaotong University(西安交通大学网络空间安全学院);School of Informatics, Xiamen University(厦门大学信息学院);National University of Singapore(新加坡国立大学)
RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Center for Excellence in Brain Science and Intelligence Technology(脑科学与智能技术卓越创新中心);University of Science and Technology of China(中国科学技术大学);Beihang University(北京航空航天大学)