追踪与协调多模态模型合并中的跨层影响
Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging
查看机构详情
- Alibaba Cloud（阿里云）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
提出TAC-Merge，通过多模态影响映射和耦合合并控制追踪并协调跨层影响，实现多模态模型合并，整合专家能力并支持泛化。
中文摘要 AI 辅助
多模态模型合并旨在将任务专家整合到一个保留其互补能力的单一模型中。大多数单模态模型合并方法在单个层内组合专家更新，而多模态方法在很大程度上遵循这一设计。然而，专家更新会改变传递给后续层的表示，使其影响能够跨深度传播，并影响视觉和文本信息交互的方式。当视觉和语言更新被组合时，后续更新作用于已被先前更新修改的输入，从而耦合其效果。这带来了两个挑战：（1）如何刻画单个专家更新在多模态中的跨深度影响，以及（2）如何基于多模态影响联合组合专家更新。为解决这些挑战，我们提出了TAC-Merge，用于在多模态模型合并中追踪和协调跨层影响。它包含两个模块，即多模态影响映射（MIM）和耦合合并控制（CMC）。MIM构建更新效果图，并利用Ricci曲率结合专家预测来定义共享融合目标。CMC对系数调整之间的交互进行建模，并联合优化区域权重以合成一个共享模型。跨多种多模态任务的实验证明了TAC-Merge在整合互补专家能力和支持对未见任务泛化方面的有效性。
英文摘要
Multimodal model merging aims to consolidate task experts into a single model that retains their complementary capabilities. Most unimodal model merging methods combine expert updates within individual layers, and multimodal approaches largely follow this design. However, an expert update changes the representations passed to subsequent layers, allowing its influence to propagate across depth and affect how visual and textual information interact. When visual and language updates are combined, later updates act on inputs already modified by earlier ones, coupling their effects. This poses two challenges: (1) how to characterize the multimodal influence of individual expert updates across depth, and (2) how to jointly combine expert updates based on their multimodal influence. To address these challenges, we propose TAC-Merge for tracing and coordinating cross-layer influence in multimodal model merging. It contains two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM constructs graphs of update effects and uses Ricci curvature together with expert predictions to define a shared fusion objective. CMC models interactions among coefficient adjustments and jointly optimizes regional weights to synthesize one shared model. Experiments across diverse multimodal tasks demonstrate the effectiveness of TAC-Merge in consolidating complementary expert capabilities and supporting generalization to unseen tasks.
