Activation-Informed Merging of Large Language Models
Massachusetts Institute of Technology(麻省理工学院);Stony Brook University(石溪大学);MIT-IBM Watson AI Lab & Red Hat AI Innovation(MIT-IBM Watson AI实验室及Red Hat AI创新)
Amin Heyrani Nobari, Kaveh Alim, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan
Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining computational efficiency. This paper introduces Activation-Informed Merging (AIM), a technique that integrates the information from the activation space of LLMs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the base model, drawing on principles from continual learning (CL) and model compression. Utilizing a task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation-space information can provide substantial advancements in the model merging strategies for LLMs, with up to a 40% increase in benchmark performance.
Activation-Guided Consensus Merging for Large Language Models
Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);City University of Hong Kong Shenzhen Research Institute(香港城市大学深圳研究院);Huawei Noah’s Ark Lab, Hong Kong SAR(华为诺亚实验室(香港特别行政区));University of Hong Kong(香港大学);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations
City University of Hong Kong, Hong Kong SAR(香港城市大学);Tsinghua University(清华大学);Huawei Noah’s Ark Lab, Hong Kong SAR(华为诺亚实验室(香港));University of Hong Kong(香港大学);Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州))
Mila – Quebec AI Institute(Mila – 魁北克人工智能研究所);Polytechnique Montréal(蒙特利尔综合理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);AI Center, Samsung, South Korea(三星人工智能中心,韩国);Samsung AI Lab, Montreal(三星人工智能实验室,蒙特利尔)
Towards Merging-conditional Backdoor Unalignment against Large Language Models
State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);Tongyi Lab, Alibaba Group Nanyang Technological University(阿里巴巴集团通义实验室南洋理工大学);City University of Hong Kong(香港城市大学)
BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generation
Department of Electrical Engineering, National Taiwan University(国立台湾大学电子工程系);Department of Computer Science & Information Engineering, National Taiwan University(国立台湾大学计算机科学与信息工程系);Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所);CSAIL, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室)
Defusing the Trigger: Tail-Risk-Informed Attention Rebalancing for LLM Backdoor Mitigation
School of Cyber Science and Technology(网络安全科学与技术学院);Harbin Institute of Technology(哈尔滨工业大学);Peng Cheng Laboratory(鹏城实验室);University of Luxembourg(卢森堡大学)
AP-BMM: Approximating Capability-Cost Pareto Sets of LLMs via Asynchronous Prior-Guided Bayesian Model Merging
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(广东新型安全智能技术重点实验室);Institute of Cyberspace Security(网络空间安全研究院);School of Computer Science and Technology(计算机科学与技术学院)