Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by precisely characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. First, we propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap within a frozen reference frame into stable biases and anisotropic residuals. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via a three-step process comprising Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable training paradigm for Multimodal Large Language Models~(MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without the need for large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.
Robust Multimodal Large Language Models Against Modality Conflict
School of Artificial Intelligence(人工智能学院);Data Science, University of Science(数据科学,科学大学);Department of Electronic Engineering(电子工程系);Information Science, University of Science(信息科学,科学大学);Huawei Technologies Co., Ltd.(华为技术有限公司)
Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Zhongke Fanyu Technology Co., Ltd(中科泛语科技有限公司)
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
School of Software, Tsinghua University, Beijing, China(清华大学软件学院,北京,中国);TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院,中国);School of New Media and Communication, Tianjin University, China(天津大学新媒体与传播学院,中国)
CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis
Information Networking Institute at Carnegie Mellon University (CMU)(卡内基梅隆大学信息网络研究所);School of Computer Science and Technology at Tongji University(同济大学计算机科学与技术学院)
Enhancing multimodal cooperation via sample-level modality valuation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));China University of Geosciences, Wuhan(中国地质大学(武汉));Nanyang Technological University(南洋理工大学);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院)
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
South China University of Technology(华南理工大学);JD Explore Academy(京东探索研究院);Nanyang Technological University(南洋理工大学);Shanghai Jiao Tong University(上海交通大学)
Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS
Computational Biomedicine Laboratory of the Foundation for Research and Technology (FORTH)(希腊国家研究与技术基金会计算生物医学实验室);Department of Electrical & Computer Engineering Hellenic Mediterranean University(希腊地中海大学电气与计算机工程系)
Deep Multimodal Learning with Missing Modality: A Survey
The Australian National University(澳大利亚国立大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Adelaide University(阿德莱德大学);The University of Surrey(萨里大学)
PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China(电子科技大学智能协同计算实验室);School of Information and Communication Engineering, University of Electronic Science and Technology of China(电子科技大学信息与通信工程学院);Peng Cheng Laboratory(鹏城实验室)
Boosting Multimodal Learning via Disentangled Gradient Learning
The Laboratory of Intelligent Collaborative Computing of UESTC(UESTC智能协同计算实验室);The School of Information and Communication Engineering of UESTC(UESTC信息与通信工程学院)
MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching
National Engineering Research Center for Multimedia Software(多媒体软件国家工程研究中心);Institute of Artificial Intelligence(人工智能研究院);School of Computer Science(计算机科学学院);Hubei Key Laboratory of Multimedia and Network Communication Engineering(湖北省多媒体与网络通信工程重点实验室);Lee Kong Chian School of Medicine(李光耀医学院);Nanyang Technological University(南洋理工大学);Ningbo Institute of Materials Technology and Engineering(宁波材料技术与工程研究所);Chinese Academy of Sciences(中国科学院);Institute for Infocomm Research (I 2 R)(信息与通信研究所以(I 2 R));Agency for Science, Technology and Research (A*STAR)(科技研究局(A*STAR))