arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.09160cs.LG
收藏

通过子空间优化实现变压器的高效资源受限训练

Efficient Resource-Constrained Training of Transformers via Subspace Optimization

LTCI, Télécom Paris, Institut Polytechnique de Paris(LTCI、巴黎电信学院、巴黎理工学院)

Le-Trung Nguyen, Enzo Tartaglione, Van-Tam Nguyen

AI总结:

本研究提出WASI方法,通过子空间优化在变压器模型中实现高效资源受限训练,显著降低内存和计算成本,提升推理效率。

摘要:

随着人工智能日益影响日常生活,能耗和数据隐私问题变得尤为突出。在设备上学习直接在边缘设备上训练模型,可以降低能耗并保护数据隐私。然而,现代神经网络的规模不断扩大,给在设备上的训练带来了重大障碍。尽管先前的工作集中在紧凑型卷积架构上,我们转而采用基于子空间的训练方法应用于变压器模型。受模型本质信息位于固定子空间的启发,我们引入了权重-激活子空间迭代(WASI)方法,该方法通过将训练限制在该子空间中,缓解了反向传播的记忆瓶颈,并通过提升变压器模型的推理效率。我们的结果表明,WASI在保持与常规训练相当的准确性的同时,将内存使用量减少了高达62倍,计算成本(FLOPs)减少了高达2倍。在Raspberry Pi 5上,WASI的训练和推理速度比常规训练快约1.4倍。代码可在https://github.com/Le-TrungNguyen/ICLR2026-WASI.git获取。

英文摘要:

As AI increasingly shapes daily life, energy consumption and data privacy have become pressing concerns. On-device learning trains models directly on edge devices, cutting energy consumption and safeguarding data privacy. However, the expanding scale of modern neural networks creates a major obstacle for on-device training. Although prior work has concentrated on compact convolutional architectures, we instead apply subspace-based training to transformer models. Motivated by the idea that a model's essential information lies in a fixed subspace, we introduce Weight-Activation Subspace Iteration (WASI), a method that mitigates the memory bottleneck of backpropagation and boosts inference efficiency in transformer models by restricting training to this subspace. Our results demonstrate that WASI maintains accuracy comparable to vanilla training while reducing memory usage by up to $62\times$ and computational cost (FLOPs) by up to $2\times$. On a Raspberry Pi 5, WASI achieves roughly $1.4\times$ faster training and inference than vanilla training. The code is available at https://github.com/Le-TrungNguyen/ICLR2026-WASI.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑