CARE-LoRA：用于内存高效LoRA的压缩激活重建
CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA
浏览论文内容
中文总结 AI 辅助
研究在有限内存下微调大型预训练模型的问题，提出CARE-LoRA框架，利用LoRA投影结构，用低秩压缩激活取代完整输入激活，并计算重建矩阵，大幅减少内存占用，性能与标准LoRA及变体相当甚至更优。
中文摘要 AI 辅助
随着大型预训练模型规模不断扩大，在有限内存预算下微调变得愈发困难。低秩适应（LoRA）是目前广泛采用的参数高效微调方法之一，通过仅优化低秩适应矩阵减轻了这一挑战，但反向传播中保留的激活成为主要内存瓶颈。为此提出CARE-LoRA，一个数据感知的压缩激活重建框架。利用LoRA的固有投影结构，用LoRA分支自然产生的低秩压缩激活取代完整输入激活，并在前向传播中计算轻量级重建矩阵，反向传播时用于重建梯度信号，使LoRA矩阵完全可训练。实验表明，CARE-LoRA大幅减少内存占用，性能与标准LoRA及变体相当甚至更优。
英文摘要
As the scale of large pre-trained models continues to grow, fine-tuning them under limited memory budgets has become increasingly challenging. Low-Rank Adaptation (LoRA), currently one of the most widely adopted parameter-efficient fine-tuning (PEFT) methods, mitigates this challenge by optimizing only low-rank adaptation matrices, thereby greatly reducing the number of trainable parameters. With the parameter overhead substantially reduced, the activations retained for backpropagation have emerged as the primary remaining memory bottleneck during LoRA fine-tuning. To address this, we propose CARE-LoRA, a data-aware Compressed Activation REconstruction framework. By exploiting the inherent projection structure of LoRA, CARE-LoRA replaces the full input activation with the low-rank compressed activation naturally produced by the LoRA branch. It further computes a lightweight reconstruction matrix during the forward pass with negligible additional computation cost, which is used during backpropagation to reconstruct the gradient signal, thereby keeping LoRA matrices fully trainable. Extensive experiments across diverse models and downstream tasks demonstrate that, while substantially reducing the overall memory footprint, CARE-LoRA achieves competitive or even superior performance compared with standard LoRA and representative LoRA variants. Our code is publicly available at https://github.com/fishandyu/CARE-LoRA .
发表机构
- Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University（上海交通大学图像处理与模式识别研究所）
机构由 AI 辅助整理，请以论文原文为准。
