各向异性模态对齐
Anisotropic Modality Align
AI总结：
本文研究了多模态模型中模态间转换的可行性，提出各向异性模态对齐方法，通过几何修正框架提升单模态数据的多模态训练效果。
摘要：
训练多模态大语言模型长期以来受到高质量配对多模态数据稀缺的限制。近期研究表明，预训练多模态对比模型的共享表示空间可以作为桥梁，使模型能够使用单模态数据进行多模态训练。然而，这一范式的关键前提仍不够明确：不同模态的表示能否可靠互换？核心障碍在于共享空间中持续存在的模态间隙。在本文中，我们重新审视模态间隙的几何本质。我们发现模态表示已经共享兼容的主导语义几何。真正阻碍模态互换性的是不是简单的全局偏移，而是一种集中在少数主导方向上的各向异性残差结构。基于这一发现，我们进一步提出各向异性模态间隙对齐原则：有效的模态对齐应与目标模态分布对齐，同时保留源模态的语义结构。受此原则指导，我们提出一个各向异性几何修正框架，AnisoAlign，用于无配对模态对齐。该框架利用目标模态的内部几何先验，并对源模态表示进行有界修正，从而在目标模态中构建替代表示。实验证实其在几何诊断和纯文本MLLM训练中的优势。总体而言，本文将模态间隙从经验观察转变为可纠正、结构化的几何现象，并为使用单模态数据训练多模态模型提供了新的表示对齐视角。
英文摘要：
Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data. Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data. However, the key premise of this paradigm remains insufficiently understood: can representations from different modalities be reliably interchanged? The core obstacle lies in the persistent Modality Gap in the shared space. In this work, we revisit the geometric nature of the modality gap. We find that modality representations already share compatible dominant semantic geometry. What truly hinders modality interchangeability is not a simple global shift, but an anisotropic residual structure concentrated along a small number of dominant directions. Based on this finding, we further propose the principle of anisotropic modality gap alignment: effective modality alignment should align with the target-modality distribution while preserving the semantic structure of the source modality. Guided by this principle, we propose an anisotropic geometric correction framework, AnisoAlign, for unpaired modality alignment. This framework leverages the internal geometric prior of the target modality and performs bounded correction on source-modality representations, thereby constructing substitute representations in the target modality. Experiments confirm its benefits in both geometric diagnostics and text-only MLLM training. Overall, this work recasts the modality gap from an empirical observation into a correctable, structured geometric phenomenon and provides a new representation alignment perspective for training multimodal models with unimodal data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
