arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.19675cs.LGcs.AI
收藏

针对内存受限微调的训练动态研究

Study of Training Dynamics for Memory-Constrained Fine-Tuning

LTCI, Télécom Paris Palaiseau(LTCI,巴黎电信Palaiseau)

Aël Quélennec, Nour Hezbri, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione

AI总结:

TraDy通过动态随机通道选择和层重要性分析,在内存受限条件下实现高效微调,提升模型性能并减少计算资源消耗。

摘要:

随着模型规模的增大,深度神经网络的内存高效训练变得越来越重要,因为部署环境对资源有严格限制。我们提出TraDy,一种新颖的迁移学习方案,利用两个关键洞察:层重要性用于更新是架构依赖的且可以预先确定,而动态随机通道选择相比静态方法提供了更优的梯度近似。我们介绍了一种动态通道选择方法,在预选层内在epoch之间随机重采样通道。广泛实验表明,TraDy在各种下游任务和架构上均实现了最先进的性能,同时保持严格的内存限制,实现了高达99%的激活稀疏性、95%的权重导数稀疏性和97%的权重导数计算FLOPs减少。

英文摘要:

Memory-efficient training of deep neural networks has become increasingly important as models grow larger while deployment environments impose strict resource constraints. We propose TraDy, a novel transfer learning scheme leveraging two key insights: layer importance for updates is architecture-dependent and determinable a priori, while dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. We introduce a dynamic channel selection approach that stochastically resamples channels between epochs within preselected layers. Extensive experiments demonstrate TraDy achieves state-of-the-art performance across various downstream tasks and architectures while maintaining strict memory constraints, achieving up to 99% activation sparsity, 95% weight derivative sparsity, and 97% reduction in FLOPs for weight derivative computation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑