用于预训练模型微调的自动编码器压缩并行分割学习
AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning
浏览论文内容
中文总结 AI 辅助
针对资源受限边缘设备上大规模基础模型分布式微调的局限，提出AE-PSL框架，利用轻量级自动编码器压缩中间激活值和梯度，并引入两阶段对齐机制，提升通信效率与模型兼容性。
中文摘要 AI 辅助
大规模基础模型在资源受限边缘设备上的分布式微调受限于本地计算约束和通信开销。并行分割学习减少了客户端计算，但客户端在每个训练步骤都要与服务器交换中间激活值和梯度。现有通信压缩方法存在不足。为此提出AE-PSL，一种通信高效的并行分割学习框架，利用位于分割层的轻量级自动编码器压缩中间激活值和梯度。为确保与预训练模型兼容，引入新颖的两阶段对齐机制，在分布式微调前使自动编码器适应预训练模型的特征流形和客户端特定特征分布。
英文摘要
Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on resource-constrained edge devices is limited by local compute constraints and communication overhead. Parallel Split Learning (PSL) reduces client-side computation by keeping few model layers on each client and offloading the remaining computation to the server; however, clients must exchange intermediate activations and gradients with the server at every training step. Existing SL communication-compression methods mainly rely on task-agnostic heuristics, such as sparsification and quantization. While learnable SL compressors can better adapt to intermediate representations, they require co-training with the target model. Therefore, directly inserting them into off-the-shelf FMs introduces feature-distribution misalignment and degrades DFT performance. To address this, we propose AE-PSL, a communication-efficient PSL framework that compresses intermediate activations and gradients using a lightweight AutoEncoder (AE) placed at the split layer. To ensure compatibility of AE compression with pre-trained FMs, AE-PSL introduces a novel two-stage alignment mechanism, which adapts the AE to the pre-trained model's feature manifold and client-specific feature distributions before DFT.
发表机构
- Eindhoven University of Technology（埃因霍温理工大学）
机构由 AI 辅助整理，请以论文原文为准。
