arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.07705cs.CL
收藏

CA-LoRA:适配现有LoRA至压缩大语言模型以在个人设备上实现高效多任务处理

CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices

Quan Cheng Laboratory(泉城实验室);Tencent Inc(腾讯公司);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)

Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, Kuai Li, Chen Chen, Tao Yang, Maosong Sun

AI总结:

本文提出CA-LoRA框架,通过知识继承与恢复策略将现有LoRA适配到压缩后的LLM,在个人设备上实现高效多任务处理,性能优于普通LoRA并接近非压缩LLM。

摘要:

近年来,在笔记本电脑和智能手机等个人设备上部署大语言模型(LLMs)的需求日益增长。这些LLM在处理不同任务时具有不同的模型变体。然而,个人设备资源有限,需要降低存储开销。为解决这一问题,现有两种关键方法:第一种是模型压缩,将LLM压缩为更小的尺寸;第二种是LoRA,它能够以极少的参数将LLM迁移到其他任务,在多任务场景中仅保留LoRA即可避免存储多个模型变体。然而,我们的实验表明,直接结合这两种方法会得到次优性能。考虑到开源社区已经为LLM贡献了大量LoRA,我们提出将这些现有LoRA从原始LLM适配到其压缩版本,并引入了一个压缩感知的LoRA(CA-LoRA)框架。我们融合了知识继承和恢复策略,以恢复因模型压缩而丢失的知识。实验结果表明,CA-LoRA优于应用于压缩LLM的普通LoRA方法,并且能够达到与带有现有LoRA模块的非压缩LLM相当的性能。CA-LoRA的源代码可在https://github.com/thunlp/CA-LoRA获取。

英文摘要:

Recently, there has been a demand to deploy Large Language Models (LLMs) on personal devices such as laptops and smartphones. These LLMs have different model variants when handling different tasks. However, personal devices have limited resources and require reduced storage overhead. To address this, there are two key methods available: the first is model compression, which compresses LLMs into smaller sizes; the second is LoRA, which can transfer an LLM to other tasks with very few parameters, avoiding the storage of multiple model variants in multi-task scenarios by only preserving LoRAs. However, our experiments show that directly combining these two methods yields sub-optimal performance. Considering that the open-source community has already contributed many LoRAs to LLMs, we propose to adapt these existing LoRAs from the LLMs to their compressed version and introduce a Compression-Aware LoRA (CA-LoRA) framework. We incorporate knowledge inheritance and recovery strategies to recover the lost knowledge caused by model compression. Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM and achieves comparable performance to the non-compressed LLM with existing LoRA modules. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑