arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.09507cs.LG
收藏

迈向多模态表示学习的统一性与对齐

Towards Uniformity and Alignment for Multimodal Representation Learning

University of Amsterdam(阿姆斯特丹大学);The Netherlands Cancer Institute(荷兰癌症研究所);The Arctic University of Norway(挪威北极大学);Singapore Management University(新加坡管理大学)

Wenzhe Yin, Pan Zhou, Zehao Xiao, Jie Liu, Shujian Yu, Jan-Jakob Sonke, Efstratios Gavves

AI总结:

本文提出了一种多模态表示学习的方法,通过分离对齐和统一性以解决冲突,提升模型在检索和生成任务中的性能。

摘要:

多模态表示学习旨在构建一个共享的嵌入空间,在其中异质模态在语义上对齐。尽管有强劲的实证结果,基于InfoNCE的目标引入了固有的冲突,导致模态之间的分布差距。在本工作中,我们识别出多模态领域中的两个冲突,这两种冲突随着模态数量的增加而加剧:(i) 一个对齐-统一性冲突,其中统一性的排斥会削弱成对对齐;(ii) 一个内在对齐冲突,其中对齐多个模态会引发竞争的对齐方向。为了解决这些问题,我们提出了一个多模态表示中对齐和统一性分离的原则性方法,提供了一个无冲突的多模态学习配方,同时支持判别性和生成性应用,而无需特定任务模块。我们随后提供了一个理论保证,表明我们的方法在多个模态分布上充当高效代理,从而减少模态之间的分布差距。在检索和UnCLIP风格生成的大量实验中,展示了持续的收益。

英文摘要:

Multimodal representation learning aims to construct a shared embedding space in which heterogeneous modalities are semantically aligned. Despite strong empirical results, InfoNCE-based objectives introduce inherent conflicts that yield distribution gaps across modalities. In this work, we identify two conflicts in the multimodal regime, both exacerbated as the number of modalities increases: (i) an alignment-uniformity conflict, whereby the repulsion of uniformity undermines pairwise alignment, and (ii) an intra-alignment conflict, where aligning multiple modalities induces competing alignment directions. To address these issues, we propose a principled decoupling of alignment and uniformity for multimodal representations, providing a conflict-free recipe for multimodal learning that simultaneously supports discriminative and generative use cases without task-specific modules. We then provide a theoretical guarantee that our method acts as an efficient proxy for a global Hölder divergence over multiple modality distributions, and thus reduces the distribution gap among modalities. Extensive experiments on retrieval and UnCLIP-style generation demonstrate consistent gains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑