基于注意力的双重压缩实现视觉Transformer的高效通信分割学习
Communication Efficient Split Learning of ViTs with Attention-based Double Compression
AI总结：
本文提出基于注意力的双重压缩（ADC）分割学习框架，通过合并相似激活和丢弃不重要token来降低ViTs中间激活的通信开销，在保持高准确率的同时优于现有SL方法。
摘要：
本文提出了一种新颖的通信高效分割学习（Split Learning, SL）框架，名为基于注意力的双重压缩（Attention-based Double Compression, ADC），旨在减少SL训练过程中传输中间视觉Transformer（Vision Transformers, ViTs）激活所需的通信开销。ADC融合了两种并行的压缩策略。第一种策略根据最后一个客户端层计算的平均注意力分数，合并相似的样本激活；该策略与类别无关，意味着它也可以合并不同类别的样本，而不会损失泛化能力或降低最终结果。第二种策略紧随第一种策略之后，丢弃最不重要的token，进一步降低通信成本。结合这两种策略不仅可以在前向传播过程中发送更少的数据，而且梯度也会被自然压缩，使得整个模型无需额外调整或对梯度进行近似即可完成训练。仿真结果表明，基于注意力的双重压缩在显著降低通信开销的同时保持了高准确率，优于最先进的SL框架。
英文摘要：
This paper proposes a novel communication-efficient Split Learning (SL) framework, named Attention-based Double Compression (ADC), which reduces the communication overhead required for transmitting intermediate Vision Transformers activations during the SL training process. ADC incorporates two parallel compression strategies. The first one merges samples' activations that are similar, based on the average attention score calculated in the last client layer; this strategy is class-agnostic, meaning that it can also merge samples having different classes, without losing generalization ability nor decreasing final results. The second strategy follows the first and discards the least meaningful tokens, further reducing the communication cost. Combining these strategies not only allows for sending less during the forward pass, but also the gradients are naturally compressed, allowing the whole model to be trained without additional tuning or approximations of the gradients. Simulation results demonstrate that Attention-based Double Compression outperforms state-of-the-art SL frameworks by significantly reducing communication overheads while maintaining high accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
