Bridging Domains through Subspace-Aware Model Merging
Universidade Estadual de Campinas (UNICAMP)(科林斯州立大学);Recod.ai Lab.(Recod.ai实验室);Instituto de Computação(计算研究所);CISPA Helmholtz Center for Information Security(信息安全赫尔姆霍茨中心);Intercom
Levy Chaves, Chao Zhou, Rebekka Burkholz, Eduardo Valle, Sandra Avila
Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task matrix into the shared basis, pruning off-diagonal components to remove conflicting singular directions. SCORE consistently outperforms, on average, existing model merging approaches in domain generalization settings across a variety of architectures and model scales, demonstrating its effectiveness and scalability.
School of Computation, Information and Technology, Technical University of Munich(计算、信息与技术学院,慕尼黑技术大学);Helmholtz Munich(海德堡-慕尼黑研究中心);Tübingen AI Center, University of Tübingen(图宾根人工智能中心,图宾根大学)
School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其跨学科应用国家重点实验室(东南大学),中华人民共和国,中国);Huawei Technologies, Shanghai, China(华为技术有限公司,上海,中国)
FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision-Language Models
College of Future Information Technology, Fudan University, Shanghai, China(复旦大学未来信息科技学院,中国);Shanghai Innovation Institute, China(上海创新研究院,中国);The Chinese University of Hong Kong, China(香港中文大学,中国);Harbin Institute of Technology, China(哈尔滨工业大学,中国);Shanghai Artificial Intelligence Laboratory, China(上海人工智能实验室,中国);Sun Yat-Sen University, Shenzhen, China(暨南大学深圳校区,中国)
SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery
Northeastern University(东北大学);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络与空间科学学院);University of Maryland(马里兰大学);Nanyang Technological University(南洋理工大学)
针对模型合并多任务学习中的表征偏差问题,本文提出 Surgery 和 SurgeryV2 模块,通过逐层对齐合并模型与专家模型的表征,显著缩小与传统多任务学习的性能差距。
Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用国家重点实验室(东南大学));Huawei Inc.(华为公司)
Bridging Training and Merging Through Momentum-Aware Optimization
Swinburne Business School, Swinburne University of Technology, Australia(斯威本科技大学斯威本商学院,澳大利亚);Data Science & Big Data Lab, Universidad Pablo de Olavide, Seville, Spain(巴勃罗·德奥拉维德大学数据科学与大数据实验室,西班牙塞维利亚)
No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Warsaw University of Technology, Poland;IDEAS Research Center, Warsaw, Poland;Department of Information Engineering, Media Integration;Communication Center (MICC), University of Florence, Italy;NASK - PIB, National Research Institute, Warsaw, Poland;Gdańsk University of Technology, Poland;Computer Vision Center, Barcelona, Spain;Department of Computer Science, Universitat Autonoma de Barcelona, Spain
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学先进交叉学科学院);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China(中国科学院人工智能安全国家重点实验室);University of Chinese Academy of Sciences, China(中国科学院大学);Alibaba Group, China(阿里巴巴集团);School of Mathematics, Southeast University, Nanjing, China(东南大学数学学院)
Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion
School of Computer Science, Wuhan University(武汉大学计算机学院);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Sun Yat-sen University(中山大学);Alibaba Group(阿里巴巴集团);School of Information and Electronics, Beijing Institute of Technology(北京理工大学信息与电子学院);Nanyang Technological University(南洋理工大学)