Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations
City University of Hong Kong, Hong Kong SAR(香港城市大学);Tsinghua University(清华大学);Huawei Noah’s Ark Lab, Hong Kong SAR(华为诺亚实验室(香港));University of Hong Kong(香港大学);Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州))
Yuxuan Yao, Haonan Sheng, Qingsong Lv, Han Wu, Shuqi Liu, Zehua Liu, Zengyan Liu, Jiahui Gao, Haochen Tan, Xiaojin Fu, Haoli Bai, Hing Cheung So, Zhijiang Guo, Linqi Song
The escalating scale of Large Language Models (LLMs) necessitates efficient adaptation techniques. Model merging has gained prominence for its efficiency and controllability. However, existing merging techniques typically serve as post-hoc refinements or focus on mitigating task interference, often failing to capture the dynamic optimization benefits of supervised fine-tuning (SFT). In this work, we propose Streaming Merging, an innovative model updating paradigm that conceptualizes merging as an iterative optimization process. Central to this paradigm is \textbf{ARM} (\textbf{A}ctivation-guided \textbf{R}otation-aware \textbf{M}erging), a strategy designed to approximate gradient descent dynamics. By treating merging coefficients as learning rates and deriving rotation vectors from activation subspaces, ARM effectively steers parameter updates along data-driven trajectories. Unlike conventional linear interpolation, ARM aligns semantic subspaces to preserve the geometric structure of high-dimensional parameter evolution. Remarkably, ARM requires only early SFT checkpoints and, through iterative merging, surpasses the fully converged SFT model. Experimental results across model scales (1.7B to 14B) and diverse domains (e.g., math, code) demonstrate that ARM can transcend converged checkpoints. Extensive experiments show that ARM provides a scalable and lightweight framework for efficient model adaptation.
Activation-Informed Merging of Large Language Models
Massachusetts Institute of Technology(麻省理工学院);Stony Brook University(石溪大学);MIT-IBM Watson AI Lab & Red Hat AI Innovation(MIT-IBM Watson AI实验室及Red Hat AI创新)
Activation-Guided Consensus Merging for Large Language Models
Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);City University of Hong Kong Shenzhen Research Institute(香港城市大学深圳研究院);Huawei Noah’s Ark Lab, Hong Kong SAR(华为诺亚实验室(香港特别行政区));University of Hong Kong(香港大学);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
Mila – Quebec AI Institute(Mila – 魁北克人工智能研究所);Polytechnique Montréal(蒙特利尔综合理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);AI Center, Samsung, South Korea(三星人工智能中心,韩国);Samsung AI Lab, Montreal(三星人工智能实验室,蒙特利尔)
AP-BMM: Approximating Capability-Cost Pareto Sets of LLMs via Asynchronous Prior-Guided Bayesian Model Merging
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(广东新型安全智能技术重点实验室);Institute of Cyberspace Security(网络空间安全研究院);School of Computer Science and Technology(计算机科学与技术学院)
Towards Merging-conditional Backdoor Unalignment against Large Language Models
State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);Tongyi Lab, Alibaba Group Nanyang Technological University(阿里巴巴集团通义实验室南洋理工大学);City University of Hong Kong(香港城市大学)
BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generation
Department of Electrical Engineering, National Taiwan University(国立台湾大学电子工程系);Department of Computer Science & Information Engineering, National Taiwan University(国立台湾大学计算机科学与信息工程系);Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所);CSAIL, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室)
Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression
School of Mathematics, Harbin Institute of Technology, Harbin, P. R. China(哈尔滨工业大学数学学院,哈尔滨,中华人民共和国);Shanghai Artificial Intelligence Laboratory, Shanghai, P. R. China(上海人工智能实验室,上海,中华人民共和国)
Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用国家重点实验室(东南大学));Huawei Inc.(华为公司)
Detecting Domain Shifts in Myoelectric Activations: Challenges and Opportunities in Stream Learning
School of Engineering(工程学院);Computer Science(计算机科学);Victoria University of Wellington(惠灵顿维多利亚大学);Artificial Intelligence Institute(人工智能研究所);University of Waikato(怀卡托大学);Department of Computer Science(计算机科学系);Big Data at Data, Intelligence and Graphs (DIG) LTCI(数据、智能与图谱(DIG)LTCI)
Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
Southern University of Science(南方科技大学);City University of Hong Kong(香港城市大学);Zhejiang University(浙江大学);Hong Kong University of Science(香港科学大学);University of Surrey(萨里大学);New York University(纽约大学)