arXiv 2604.228232026-07-14cs.CVcs.AI

PivotMerge: 通过后对齐模型融合弥合异构多模态预训练

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Data Science and Artificial Intelligence Research Institute, China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司数据科学与人工智能研究院）

Zibo Shao, Baochen Xiong, Xiaoshan Yang, Yaguang Song, Qimeng Zhang, Haifeng Chen, Changsheng Xu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出PivotMerge框架，通过后对齐模型融合技术，解决多模态预训练中跨模态对齐能力整合问题，提升模型统一语义空间的构建效果。

摘要：

多模态大语言模型（MLLMs）依赖于从多样数据源进行多模态预训练，其中不同数据集往往诱导出互补的跨模态对齐能力。模型融合提供了一种成本效益高的机制，用于将多个具有互补优势的MLLMs整合成一个统一的模型。然而，现有模型融合研究主要集中在后微调场景，预训练阶段鲜有探索。我们认为，MLLMs预训练的核心在于建立有效的跨模态对齐，将视觉和文本表示整合到统一的语义空间中。受此启发，我们引入了后对齐融合任务，旨在整合从异构多模态预训练中学习到的跨模态对齐能力。这一设置引入了两个关键挑战：跨域参数干扰，即在融合过程中，从不同数据分布学习的参数更新会冲突；以及层间对齐贡献差异，即不同层和投影器对跨模态对齐的贡献不均。为解决这些问题，我们提出了PivotMerge，一种用于跨模态投影器的后对齐融合框架。PivotMerge包含两个关键组件：共享空间分解与过滤，将共享对齐模式与领域特定变化解耦，并抑制冲突方向；以及对齐引导的层间融合，根据不同的对齐贡献分配层特定的融合权重。我们构建了基于CC12M的系统后对齐融合场景进行评估。在多个多模态基准上的广泛实验表明，PivotMerge在多个多模态基准上持续优于现有基线，证明了其有效性与泛化能力。

英文摘要：

Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities. Model merging provides a cost-effective mechanism for integrating multiple expert MLLMs with complementary strengths into a unified model. However, existing model merging research mainly focuses on post-finetuning scenarios, leaving the pre-training stage largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from heterogeneous multimodal pre-training. This setting introduces two key challenges: cross-domain parameter interference, where parameter updates learned from different data distributions conflict during merging, and layer-wise alignment contribution disparity, where different layers and projectors contribute unevenly to cross-modal alignment. To address them, we propose \textbf{PivotMerge}, a post-alignment merging framework for cross-modal projectors. PivotMerge incorporates two key components: Shared-space Decomposition and Filtering, which disentangles shared alignment patterns from domain-specific variations and suppresses conflicting directions, and Alignment-guided Layer-wise Merging, which assigns layer-specific merging weights based on differing alignment contributions. We construct systematic CC12M-based post-alignment merging scenarios for evaluation. Extensive experiments on multiple multimodal benchmarks show that PivotMerge consistently outperforms existing baselines, demonstrating its effectiveness and generalization ability.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。