arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.07026cs.CVcs.AIcs.MM
收藏

模态间隙驱动的子空间对齐训练范式用于多模态大语言模型

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

HKUST(GZ)(香港科技大学(广州));NUS(新加坡国立大学);sh AILab;SII;Stanford(斯坦福大学);UCLA(加州大学洛杉矶分校);Yale(耶鲁大学);SJTU(上海交通大学);GBU(国防大学);PKU(北京大学)

Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

AI总结:

针对多模态对比学习中的模态间隙问题,提出固定帧模态间隙理论,并基于该理论设计无训练的对齐策略ReAlign和可扩展训练范式ReVision,利用无配对数据实现视觉与语言表示的高效对齐。

摘要:

尽管多模态对比学习在视觉和语言表示对齐方面取得了成功,但一个持久的几何异常——模态间隙——仍然存在:表达相同语义的不同模态的嵌入位于系统性偏移的区域。先前弥合这一间隙的方法大多受限于过于简化的各向同性假设,阻碍了它们在大规模场景中的应用。在本文中,我们通过精确刻画模态间隙的几何形状并利用它进行高效模型扩展来解决这些局限性。首先,我们提出了固定帧模态间隙理论,该理论将冻结参考帧内的模态间隙分解为稳定偏差和各向异性残差。在这种精确建模的指导下,我们引入了ReAlign,一种无需训练的模态对齐策略。利用大量无配对数据的统计信息,ReAlign通过锚点、轨迹和质心对齐三步过程将文本表示对齐到图像表示分布,从而显式纠正几何错位。基于ReAlign,我们提出了ReVision,一种用于多模态大语言模型(MLLMs)的可扩展训练范式。ReVision将ReAlign集成到预训练阶段,使模型在视觉指令微调之前从无配对文本中学习视觉表示的分布,无需大规模、高质量的图像-文本对。我们的框架表明,统计对齐的无配对数据可以有效替代昂贵的图像-文本对,为MLLMs的高效扩展提供了一条稳健的路径。

英文摘要:

Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by precisely characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. First, we propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap within a frozen reference frame into stable biases and anisotropic residuals. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via a three-step process comprising Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable training paradigm for Multimodal Large Language Models~(MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without the need for large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑