Checkpoint merging is a technique for combining multiple model snapshots into a single superior model, potentially reducing training time for large language models. This paper explores checkpoint merging in the context of parameter-efficient fine-tuning (PEFT), where only small adapter modules (e.g. LoRA) are trained. We propose Metrics-Weighted Averaging (MWA), a simple yet effective method to merge model checkpoints by weighting their parameters according to performance metrics. In particular, we investigate weighting by training loss and by training steps, under the intuition that lower-loss or later-step checkpoints are more valuable. We introduce a formula with a penalty factor to adjust weight distribution, requiring only one hyperparameter regardless of the number of checkpoints. Experiments on three fine-tuning tasks (mathematical reasoning, preference alignment, and general instruction tuning) show that MWA consistently produces merged models that outperform the naive uniform average of checkpoints. Notably, loss-weighted merging often yields the best results, delivering up to 5% higher task accuracy than the baseline uniform merge and even surpassing the final individual checkpoint's performance. These findings validate checkpoint merging for PEFT and demonstrate that a metric-driven weighting heuristic can efficiently boost model performance with minimal computational overhead.
Generalizing the Geometry of Model Merging Through Frechet Averages
Faculty of Computer Science, Dalhousie University(达尔豪斯大学计算机科学学院);Vector Insitute for Artificial Intelligence(向量人工智能研究所);Schulich School of Engineering, University of Calgary(卡莱尔大学谢尔奇工程学院);Department of Computer Science and Software Engineering, Concordia University(康科迪亚大学计算机科学与软件工程系);Mila - Quebec AI Institute(魁北克人工智能研究所)
Reliability-Aware Checkpoint Selection for Domain Generalization
Shenzhen University(深圳大学);Xiamen University(厦门大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Macao Polytechnic University(澳门理工大学);Fudan University(复旦大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
MagMax: Leveraging Model Merging for Seamless Continual Learning
IDEAS NCBR;Warsaw University of Technology(华沙工业大学);Autonomous University of Barcelona(巴塞罗那自治大学);Computer Vision Center(计算机视觉中心);Tooploox;Gdańsk University of Technology(格但斯克工业大学)
QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency
University of Wisconsin – Madison(威斯康星大学麦迪逊分校);Baidu Inc.(百度公司);University of Arizona(亚利桑那大学);University of Toronto(多伦多大学);Wilfrid Laurier University(威尔弗里德·劳里埃大学);York University(约克大学)
SeWA: Selective Weight Average via Probabilistic Masking
Huazhong University of Science and Technology(华中科技大学);Shanghai Jiao Tong University(上海交通大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Baidu Inc.(百度公司);Sun Yat-sen University(中山大学);Nanyang Technological University(南洋理工大学)
Model Merging by Uncertainty-Based Gradient Matching
Technical University of Darmstadt(达姆施塔特工业大学);Hessian Center for AI (hessian.AI)(黑森人工智能中心);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);University of Edinburgh(爱丁堡大学)
Trainable Weight Averaging: Accelerating Training and Improving Generalization
Shanghai Jiao Tong University(上海交通大学);KU Leuven(鲁汶大学);National Taiwan University(台湾大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Weight Scope Alignment: A Frustratingly Easy Method for Model Merging
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学先进交叉学科学院);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China(中国科学院人工智能安全国家重点实验室);University of Chinese Academy of Sciences, China(中国科学院大学);Alibaba Group, China(阿里巴巴集团);School of Mathematics, Southeast University, Nanjing, China(东南大学数学学院)
A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs
Deakin Cyber Research(德肯网络研究);Innovation Centre Deakin University / Universal Higher Education Melbourne, Australia(德肯大学创新中心/通用高等教育墨尔本,澳大利亚);School of IT(信息科技学院);Engineering Melbourne Institute of Technology Melbourne, Australia(工程墨尔本技术学院墨尔本,澳大利亚);Innovation Centre Deakin University Geelong, Australia(德肯大学创新中心吉朗,澳大利亚)