arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.02421cs.CLcs.AI
收藏

大语言模型的激活信息引导合并

Activation-Informed Merging of Large Language Models

Massachusetts Institute of Technology(麻省理工学院);Stony Brook University(石溪大学);MIT-IBM Watson AI Lab & Red Hat AI Innovation(MIT-IBM Watson AI实验室及Red Hat AI创新)

Amin Heyrani Nobari, Kaveh Alim, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan

AI总结:

本文提出激活信息引导合并方法AIM,利用LLM激活空间信息在合并时选择性保留关键权重,可适配任意现有合并方法,并在多个基准上实现最高40%的性能提升。

摘要:

模型合并是一种将多个微调后的大语言模型(LLMs)的参数和嵌入进行组合的方法,为在保持计算效率的同时提升模型在各种任务上的性能提供了一种有前景的途径。本文提出了激活信息引导合并(Activation-Informed Merging,AIM),一种将LLMs激活空间中的信息整合到合并过程中的技术,以提升性能和鲁棒性。AIM被设计为一种灵活的、互补的解决方案,适用于任何现有的合并方法。它借鉴持续学习(CL)和模型压缩的原理,旨在保留基础模型中的关键权重。利用一个与任务无关的校准集,AIM在合并过程中选择性地优先处理重要权重。我们通过实验证明,AIM在多个基准测试上显著提升了合并模型的性能。我们的研究结果表明,考虑激活空间信息可以为LLMs的模型合并策略带来实质性进步,基准测试性能提升最高可达40%。

英文摘要:

Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining computational efficiency. This paper introduces Activation-Informed Merging (AIM), a technique that integrates the information from the activation space of LLMs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the base model, drawing on principles from continual learning (CL) and model compression. Utilizing a task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation-space information can provide substantial advancements in the model merging strategies for LLMs, with up to a 40% increase in benchmark performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑