SplitLite：面向分裂学习的低秩残差压缩
SplitLite: Low-Rank Residual Compression for Split Learning
浏览论文内容
中文总结 AI 辅助
本文针对设备端LLM联邦微调的高通信成本问题，提出SplitLite方法，利用残差的低秩结构压缩传输量，在保持性能的同时大幅降低了通信成本。
中文摘要 AI 辅助
针对设备端大语言模型（LLM）的联邦微调面临显著的计算负担，分裂学习（SL）作为一种有前景的解决方案应运而生，它将主要训练工作负载卸载到强大的服务器上。然而，SL需要在客户端与服务器之间交换高维激活值和梯度，导致通信成本过高。为克服这一挑战，本文提出SplitLite，一种通信高效的分裂联邦LoRA微调方法，该方法利用连续轮次激活值和梯度残差的低有效秩结构。我们的关键发现是，当LoRA在参数空间使用秩r的更新时，同一数据样本在相邻轮次间的激活值残差和梯度残差分别呈现有效秩-2r和秩-4r的结构。通过揭示这一特性，SplitLite仅传输量化截断奇异值分解（SVD）残差因子，从而显著降低激活值上行链路和梯度下行链路的通信量。在GLUE基准上针对一系列先进设备端LLM开展的大量实验表明，我们的方法可将激活值上行链路通信成本降低多达93.5%，总通信成本降低多达83.7%，且不会出现性能下降。
英文摘要
Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL) has emerged as a promising solution, which offloads the primary training workload to a powerful server. However, SL requires exchanging high-dimensional activations and gradients between clients and the server, resulting in prohibitive communication costs. To overcome this challenge, we propose SplitLite, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals. Our key finding is that, when LoRA uses rank $r$ updates in parameter space, the activation and gradient residuals of the same data sample between adjacent epochs also exhibit effective rank-$2r$ and rank-$4r$ structures, respectively. By revealing this property, SplitLite transmits only quantized truncated singular value decomposition (SVD) residual factors, thereby significantly reducing both activation uplink and gradient downlink traffic. Extensive experiments on the GLUE benchmark across a series of advanced on-device LLMs demonstrate that our method reduces activation uplink communication costs by up to 93.5\% and total communication costs by up to 83.7\%, without performance degradation.
发表机构
- The University of Hong Kong（香港大学）
- California Institute of Technology（加州理工学院）
机构由 AI 辅助整理，请以论文原文为准。
