arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.05768cs.LGcs.AIcs.CV
收藏

通过子空间感知模型融合跨领域

Bridging Domains through Subspace-Aware Model Merging

Universidade Estadual de Campinas (UNICAMP)(科林斯州立大学);Recod.ai Lab.(Recod.ai实验室);Instituto de Computação(计算研究所);CISPA Helmholtz Center for Information Security(信息安全赫尔姆霍茨中心);Intercom

Levy Chaves, Chao Zhou, Rebekka Burkholz, Eduardo Valle, Sandra Avila

AI总结:

本文提出SCORE方法,通过子空间冲突缓解技术提升模型融合在跨领域泛化中的性能。

摘要:

模型融合将多个任务特定模型整合为一个统一的模型。最近的研究在改进分布内或多任务场景下的融合性能方面取得了进展,但模型融合中的领域泛化仍处于探索阶段。我们研究了在不同领域上微调的融合模型对未见领域泛化的影响。通过使用奇异值分解分析任务矩阵中的参数竞争,我们发现在不同分布偏移下训练的融合模型其子空间之间冲突比传统多任务设置更强。为缓解这一问题,我们提出了SCORE(子空间冲突缓解融合)方法,旨在缓解此类奇异子空间冲突。SCORE通过计算所有模型拼接后的领先奇异向量的主成分,找到共享的正交基。然后将每个任务矩阵投影到共享基中,剪除非对角成分以去除冲突的奇异方向。在多种架构和模型规模的领域泛化设置中,SCORE在平均上优于现有模型融合方法,证明了其有效性和可扩展性。

英文摘要:

Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task matrix into the shared basis, pruning off-diagonal components to remove conflicting singular directions. SCORE consistently outperforms, on average, existing model merging approaches in domain generalization settings across a variety of architectures and model scales, demonstrating its effectiveness and scalability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑