arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17635cs.CLcs.AIcs.CV
收藏

上下文元LoRA生成

In-Context Meta LoRA Generation

The Hong Kong Polytechnic University(香港理工大学);Tsinghua University(清华大学);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院);University of Trento(特伦托大学);University of Copenhagen(哥本哈根大学);Peking University(北京大学)

Yihua Shao, Minxi Yan, Yang Liu, Siyu Chen, Wenjie Chen, Xinwei Long, Ziyang Yan, Lei Li, Chenyu Zhang, Nicu Sebe, Hao Tang, Yan Wang, Hao Zhao, Mengzhu Wang, Jingcai Guo

AI总结:

本文提出ICM-LoRA,利用条件变分自编码器结合上下文元学习,从任务描述直接生成任务感知的LoRA参数,实现多任务高效定制,仅占原始LoRA 1%的存储。

摘要:

低秩适应(LoRA)在任务特定微调中展现了卓越的能力。然而,在涉及多个任务的场景中,为每个任务分别训练一个LoRA模型会在存储和推理方面造成相当大的低效。此外,现有的参数生成方法未能捕捉这些任务之间的相关性,使得多任务LoRA参数生成具有挑战性。为了解决这些局限性,我们提出了上下文元LoRA(ICM-LoRA),这是一种高效实现大语言模型(LLMs)任务特定定制的新方法。具体来说,我们使用所有任务的训练数据来训练一个定制生成器,即条件变分自编码器(CVAE)。CVAE以任务描述为输入,并生成任务感知的LoRA权重作为输出。这些LoRA权重随后与LLMs合并,以创建任务专用模型,而无需额外的微调。此外,我们利用上下文元学习进行知识增强和任务映射,以捕捉任务与参数分布之间的关系。因此,我们的方法利用CVAE为多样任务实现了更准确的LoRA参数生成。ICM-LoRA能够比当前的参数重建方法实现更准确的LoRA参数重建,并且有助于实现LoRA参数的任务特定增强。同时,我们的方法占用283MB,仅为原始LoRA存储的1%。

英文摘要:

Low-rank Adaptation (LoRA) has demonstrated remarkable capabilities for task specific fine-tuning. However, in scenarios that involve multiple tasks, training a separate LoRA model for each one results in considerable inefficiency in terms of storage and inference. Moreover, existing parameter generation methods fail to capture the correlations among these tasks, making multi-task LoRA parameter generation challenging. To address these limitations, we propose In-Context Meta LoRA (ICM-LoRA), a novel approach that efficiently achieves task-specific customization of large language models (LLMs). Specifically, we use training data from all tasks to train a tailored generator, Conditional Variational Autoencoder (CVAE). CVAE takes task descriptions as inputs and produces task-aware LoRA weights as outputs. These LoRA weights are then merged with LLMs to create task-specialized models without the need for additional fine-tuning. Furthermore, we utilize in-context meta-learning for knowledge enhancement and task mapping, to capture the relationship between tasks and parameter distributions. As a result, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE. ICM-LoRA enables more accurate LoRA parameter reconstruction than current parameter reconstruction methods and is useful for implementing task-specific enhancements of LoRA parameters. At the same time, our method occupies 283MB, only 1\% storage compared with the original LoRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑