仅一次通信：边缘设备上的通信高效拆分式联邦大语言模型微调
Just Talk Once: Communication-Efficient Split Federated LLM Fine-Tuning on Edge Devices
浏览论文内容
中文总结 AI 辅助
针对边缘设备LLM微调的通信与客户端在线时间瓶颈，提出L型SFT及基于它的一次性SFT框架，实现仅一次通信即可完成微调，显著降低了通信成本和客户端在线时间。
中文摘要 AI 辅助
大语言模型（LLM）微调正越来越多地转向边缘设备生成的数据，而边缘设备的内存、计算、带宽和连接性限制使得传统联邦学习难以持续。拆分式联邦微调（SFT）通过将大部分模型参数和计算任务卸载到服务器来提高客户端效率，但它需要在拆分接口处进行逐步双向通信循环，并迫使客户端在整个训练过程中持续参与。本文提出L型SFT，这一拆分式微调框架消除了该双向瓶颈。核心见解是，现代LLM中的权重绑定使服务器端隐藏激活能够使用目标嵌入直接进行监督，从而无需将服务器输出返回给客户端即可在服务器上计算训练损失。为进一步消除对客户端持续参与的需求，基于L型SFT，我们引入一次性SFT，其中客户端仅上传一次激活，之后即可离线，而服务器则继续对缓存的表示进行优化。我们在包含商业智能手机和NVIDIA开发板等异构边缘客户端的真实系统测试平台中实现了该设计。实验表明，与现有SFT基线相比，我们的方案显著降低了通信成本和客户端在线时间。
英文摘要
Large language model (LLM) fine-tuning is increasingly shifting toward data generated on edge devices, where memory, computation, bandwidth, and connectivity constraints make conventional federated learning difficult to sustain. Split federated fine-tuning (SFT) improves client-side efficiency by offloading most model parameters and computation to the server but requires step-by-step bidirectional communication loop across the split interface and forces continuous client involvement throughout training. In this paper, we present L-shaped SFT, a split fine-tuning framework that removes this bidirectional bottleneck. Our key insight is that weight tying in modern LLMs enables server-side hidden activations to be directly supervised using target embeddings, allowing the training loss to be computed on the server without returning server outputs to the client. To further eliminate the need for continuous client participation, based on L-shaped SFT, we introduce one-shot SFT, in which clients upload activations once and then go offline while the server continues optimization over cached representations. We implement our design in a real system testbed with heterogeneous edge clients, including commercial smartphones and NVIDIA developer boards. Experiments demonstrate that our schemes significantly reduce communication costs and client online time compared with existing SFT baselines.
发表机构
- Duke Kunshan University（杜克昆山大学）
- The University of Hong Kong（香港大学）
机构由 AI 辅助整理，请以论文原文为准。
