Towards Uniformity and Alignment for Multimodal Representation Learning
University of Amsterdam(阿姆斯特丹大学);The Netherlands Cancer Institute(荷兰癌症研究所);The Arctic University of Norway(挪威北极大学);Singapore Management University(新加坡管理大学)
Wenzhe Yin, Pan Zhou, Zehao Xiao, Jie Liu, Shujian Yu, Jan-Jakob Sonke, Efstratios Gavves
Multimodal representation learning aims to construct a shared embedding space in which heterogeneous modalities are semantically aligned. Despite strong empirical results, InfoNCE-based objectives introduce inherent conflicts that yield distribution gaps across modalities. In this work, we identify two conflicts in the multimodal regime, both exacerbated as the number of modalities increases: (i) an alignment-uniformity conflict, whereby the repulsion of uniformity undermines pairwise alignment, and (ii) an intra-alignment conflict, where aligning multiple modalities induces competing alignment directions. To address these issues, we propose a principled decoupling of alignment and uniformity for multimodal representations, providing a conflict-free recipe for multimodal learning that simultaneously supports discriminative and generative use cases without task-specific modules. We then provide a theoretical guarantee that our method acts as an efficient proxy for a global Hölder divergence over multiple modality distributions, and thus reduces the distribution gap among modalities. Extensive experiments on retrieval and UnCLIP-style generation demonstrate consistent gains.
Graph-based Alignment and Uniformity for Recommendation
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Salesforce AI Research(赛富时人工智能研究院);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);University of Electronic Science and Technology of China(电子科技大学)
DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Beijing National Research Center for Information Science and Technology, Tsinghua University(清华大学信息科学与技术国家研究中心);Department of Automation, Tsinghua University(清华大学自动化系)
PSQE: A Theoretical-Practical Approach to Pseudo Seed Quality Enhancement for Unsupervised Multimodal Entity Alignment
Key Laboratory of Knowledge Engineering with Big Data (the Ministry of Education of China), Hefei University of Technology(大数据知识工程重点实验室(教育部)、合肥工业大学);Department of Data Science, College of William and Mary(威廉与玛丽学院数据科学系);College of Computer and Information Science, Southwest University(西南大学计算机与信息科学学院)
InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Shanghai Jiao Tong University(上海交通大学);Hong Kong University of Science and Technology(香港科技大学)
LUMA-RAG: Lifelong Multimodal Agents with Provably Stable Streaming Alignment
Dept. of Computer Engineering(计算机工程系);SIES Graduate School of Technology(SIES技术研究生学院);Bharatiya Vidya Bhavan's Sardar Patel Institute of Technology(巴哈里亚·维达·巴万学院萨达尔·帕特尔技术学院)
Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation
College of Computer and Mathematics, Central South University of Forestry and Technology(中南林业科技大学计算机与数学学院);Department of Computer Science, State University of New York(纽约州立大学计算机科学系)
LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation
Department of Machine Learning, CMU, USA(卡内基梅隆大学机器学习系,美国);Department of Machine Learning, MBZUAI, UAE(穆罕默德·本·扎耶德人工智能大学机器学习系,阿联酋);Department of Artificial Intelligence, Hanyang University ERICA, South Korea(汉阳大学ERICA校区人工智能系,韩国)
MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features
Karlsruhe University of Applied Sciences(卡尔斯鲁厄应用技术大学);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);University of Hohenheim(霍恩海姆大学)
MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion
University of Shanghai for Science and TechnologyChina(上海科学技术大学中国);Shenzhen Institute of Advanced Technology, Chinese Academy of SciencesChina(深圳先进技术研究院,中国科学院中国);University of Minnesota-Twin Cities, USA(明尼苏达大学双城分校,美国);University of LeedsUK(利兹大学,英国)