NOLA:利用随机基的线性组合压缩 LoRA
NOLA: Compressing LoRA using Linear Combination of Random Basis
AI总结:
NOLA 通过随机基的线性组合重新参数化 LoRA 低秩矩阵,仅优化混合系数,从而突破秩一下界,在保持精度的同时大幅压缩可训练参数,在 LLaMA-2 70B 上比最压缩的 LoRA 紧凑近 20 倍。
摘要:
由于模型规模巨大(例如 GPT-3 为 350GB),针对每个下游任务或领域微调并存储大语言模型(LLMs)是不切实际的。现有文献(如 LoRA)展示了对 LLM 原始权重进行低秩修改的潜力,从而能够高效适配和存储任务特定模型。这些方法可以将微调 LLM 所需的参数数量减少几个数量级。然而,这些方法面临两个主要局限:(1)参数数量受到秩一分解的下界限制;(2)压缩程度在很大程度上受模型架构和所选秩的影响。我们提出 NOLA,它克服了 LoRA 中存在的秩一下界。NOLA 通过使用随机生成矩阵(基)的线性组合对 LoRA 中的低秩矩阵进行重新参数化,并仅优化线性混合系数来实现这一目标。这种方法使我们能够将可训练参数的数量与秩的选择和网络架构解耦。我们展示了使用 GPT-2、LLaMA-2 和 ViT 在自然语言和计算机视觉任务上的适配结果。与秩一的 LoRA(即 LoRA 所能达到的最佳压缩)相比,NOLA 在使用少得多的参数的情况下表现与 LoRA 模型相当。特别是在 LLaMA-2 70B 上,我们的方法比最压缩的 LoRA 紧凑近 20 倍,且准确率没有下降。我们的代码可在以下地址获取:https://github.com/UCDvision/NOLA
英文摘要:
Fine-tuning Large Language Models (LLMs) and storing them for each downstream task or domain is impractical because of the massive model size (e.g., 350GB in GPT-3). Current literature, such as LoRA, showcases the potential of low-rank modifications to the original weights of an LLM, enabling efficient adaptation and storage for task-specific models. These methods can reduce the number of parameters needed to fine-tune an LLM by several orders of magnitude. Yet, these methods face two primary limitations: (1) the parameter count is lower-bounded by the rank one decomposition, and (2) the extent of reduction is heavily influenced by both the model architecture and the chosen rank. We introduce NOLA, which overcomes the rank one lower bound present in LoRA. It achieves this by re-parameterizing the low-rank matrices in LoRA using linear combinations of randomly generated matrices (basis) and optimizing the linear mixture coefficients only. This approach allows us to decouple the number of trainable parameters from both the choice of rank and the network architecture. We present adaptation results using GPT-2, LLaMA-2, and ViT in natural language and computer vision tasks. NOLA performs as well as LoRA models with much fewer number of parameters compared to LoRA with rank one, the best compression LoRA can archive. Particularly, on LLaMA-2 70B, our method is almost 20 times more compact than the most compressed LoRA without degradation in accuracy. Our code is available here: https://github.com/UCDvision/NOLA
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。