AdaMerging: Adaptive Model Merging for Multi-Task Learning
Northeastern University（东北大学）；University of Oxford（牛津大学）；Nanyang Technological University（南洋理工大学）；University of Maryland（马里兰大学）；JD Explore Academy（京东探索研究院）
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, Dacheng Tao
Multi-task learning (MTL) aims to empower a model to tackle multiple tasks simultaneously. A recent development known as task arithmetic has revealed that several models, each fine-tuned for distinct tasks, can be directly merged into a single model to execute MTL without necessitating a retraining process using the initial training data. Nevertheless, this direct addition of models often leads to a significant deterioration in the overall performance of the merged model. This decline occurs due to potential conflicts and intricate correlations among the multiple tasks. Consequently, the challenge emerges of how to merge pre-trained models more effectively without using their original training data. This paper introduces an innovative technique called Adaptive Model Merging (AdaMerging). This approach aims to autonomously learn the coefficients for model merging, either in a task-wise or layer-wise manner, without relying on the original training data. Specifically, our AdaMerging method operates as an automatic, unsupervised task arithmetic scheme. It leverages entropy minimization on unlabeled test samples from the multi-task setup as a surrogate objective function to iteratively refine the merging coefficients of the multiple models. Our experimental findings across eight tasks demonstrate the efficacy of the AdaMerging scheme we put forth. Compared to the current state-of-the-art task arithmetic merging scheme, AdaMerging showcases a remarkable 11\% improvement in performance. Notably, AdaMerging also exhibits superior generalization capabilities when applied to unseen downstream tasks. Furthermore, it displays a significantly enhanced robustness to data distribution shifts that may occur during the testing phase.
The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging
School of Computation, Information and Technology, Technical University of Munich, Germany（计算、信息与技术学院，慕尼黑技术大学）；Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich, Germany（生物医学影像机器学习研究所，海德堡慕尼黑研究中心）；relAI – Konrad Zuse School of Excellence in Reliable AI（relAI – 卡诺尔·祖斯可靠人工智能卓越学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；School of Biomedical Engineering and Imaging Sciences, King's College London, UK（生物医学工程与成像科学学院，伦敦国王学院）；Institute of Pathology, Technical University of Munich, Germany（病理学研究所，慕尼黑技术大学）
Interpretable Diagnostics and Adaptive Data Assimilation for Neural ODEs via Discrete Empirical Interpolation
School of Mechanical Engineering, Purdue University（普渡大学机械工程学院）；Mathematics and Computer Science Division, Argonne National Laboratory（阿贡国家实验室数学与计算机科学部）
Adaptive Pareto-Optimal Token Merging for Edge Transformer Models in Semantic Communication
Department of Electrical Engineering（电气工程系）；KU 6G Research Centre, College of Computing（KU 6G研究中心，计算学院）；Mathematical Sciences, Khalifa University, UAE（数学科学，卡利法大学，阿联酋）；Department of Electrical and Software Engineering（电气与软件工程系）；University of Calgary, Canada（卡尔加里大学，加拿大）；Centre for Wireless Communications, University of Oulu, Finland（无线通信中心，奥卢大学，芬兰）
Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
Wuhan University（武汉大学）；Northeastern University（东北大学）；Sun Yat-sen University（中山大学）；Beijing Institute of Technology（北京理工大学）；Nanyang Technological University（南洋理工大学）
Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration
Key Laboratory of Big Data & Artificial Intelligence in Transportation, Beijing, China（大数据与人工智能交通实验室，北京，中国）；School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（北京交通大学计算机科学与技术学院，北京，中国）；Frontiers Science Center for Smart High-Speed Railway System, Beijing Jiaotong University, Beijing, China（智能高速铁路系统前沿科学中心，北京交通大学，北京，中国）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院，新加坡）
Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用国家重点实验室（东南大学））；Huawei Inc.（华为公司）
Coordinated Lane-Level Variable Speed Limits and Ramp Metering for Successive Weaving Segments Considering Merging/Diverging Risks: A Hybrid Model Predictive Control and Multi-Agent Reinforcement Learning Approach
