DIRO & Institut Courtois, Université de Montréal(迪罗与库尔托伊研究所,蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);Université Sorbonne Paris Nord, LIPN, CNRS(索邦巴黎北大学,LIPN,CNRS);Université Paris Cité, LLF, CNRS(巴黎城市大学,LLF,CNRS);Canada CIFAR AI Chair(加拿大CIFAR人工智能 chair)
Huan Zhang, Yanjian Zhang, Guillaume Wisniewski, Nadi Tomeh, Bang Liu
We propose a sparsity-aware evolutionary (SAE) framework for model merging that involves iterative pruning-merging cycles to act as a novel mutation operator. We incorporate the sparsity constraints into the score function, which steers the evolutionary process to favor more sparse models, in addition to other conventional performance scores. Interestingly, the by-product of \textit{competition} for sparsity introduces an extra local \textit{attraction} and interplay into the evolutionary process: if one competitor has more zero elements, the other competitor's non-zero elements will occupy those positions, even though the less sparse competitor loses to the more sparse competitor in other positions. The proposed pipeline is evaluated on a variety of large-scale LLM benchmarks. Experiments demonstrate that our approach can improve model merging reliability across multiple benchmarks, and is easy to incorporate due to its simplicity and being orthogonal to most existing approaches.
LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models
School of Microelectronics, Southern University of Science and Technology(微电子学院,南方科技大学);Department of Electrical and Electronic Engineering, University of Hong Kong(电气与电子工程系,香港大学)
Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging
Zhejiang University(浙江大学);Simon Fraser University(西蒙菲莎大学);The Chinese University of Hong Kong(香港中文大学);Zhejiang Key Lab of Accessible Perception and Intelligent Systems(浙江省可感知智能系统重点实验室)
CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Institute for Artificial Intelligence, Tsinghua University (THUAI)(人工智能研究所,清华大学);Beijing National Research Center for Information Science and Technology (BNRist)(北京国家信息科学与技术研究中心);Department of Automation, Tsinghua University(自动化系,清华大学);East China Normal University (ECNU)(华东师范大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算机与信息系统学院)
Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用国家重点实验室(东南大学));Huawei Inc.(华为公司)
Bridging Domains through Subspace-Aware Model Merging
Universidade Estadual de Campinas (UNICAMP)(科林斯州立大学);Recod.ai Lab.(Recod.ai实验室);Instituto de Computação(计算研究所);CISPA Helmholtz Center for Information Security(信息安全赫尔姆霍茨中心);Intercom
Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization
University of Amsterdam(阿姆斯特丹大学);Informatics Institute, University of Amsterdam(阿姆斯特丹大学信息学院);Amsterdam Machine Learning Lab (AMLab), University of Amsterdam(阿姆斯特丹大学机器学习实验室)
Bridging Training and Merging Through Momentum-Aware Optimization
Swinburne Business School, Swinburne University of Technology, Australia(斯威本科技大学斯威本商学院,澳大利亚);Data Science & Big Data Lab, Universidad Pablo de Olavide, Seville, Spain(巴勃罗·德奥拉维德大学数据科学与大数据实验室,西班牙塞维利亚)
Faculty of Mathematics and Informatics, Hanoi University of Science and Technology(河内科学技术大学数学与信息学院);Viettel AI, Viettel Group(Viettel集团Viettel人工智能公司);Department of Mathematics, National University of Singapore(新加坡国立大学数学系)
Study of Sparsity-Aware Subband Adaptive Filtering Algorithms with Adjustable Penalties
Southwest University of Science and Technology(西南科技大学);Southwest Jiaotong University(西南交通大学);PUC-Rio(里约热内卢天主教大学);University of York(约克大学);Sichuan University(四川大学)
SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution
School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);University of California San Diego(加州大学圣地亚哥分校)
Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inference
National Key Research and Development Program of China(国家重点研发计划);Shenzhen Science and Technology Program(深圳科技计划);National Natural Science Foundation of China(国家自然科学基金)
Sparsity-Aware SSAF Algorithm with Individual Weighting Factors for Acoustic Echo Cancellation
Southwest University of Science and Technology(西南科技大学);Zhejiang Lab(之江实验室);PUC-Rio(里约热内卢天主教大学);University of York(约克大学);Harbin Engineering University(哈尔滨工程大学)