arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07332cs.CV
收藏

最小化深度学习模型训练中的能源成本:高斯采样方法

Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approach

Indian Institute of Technology Hyderabad(印度理工学院海得拉巴分校)

Challapalli Phanindra Revanth, Sumohana S. Channappayya, C Krishna Mohan

AI总结:

针对反向传播训练能耗高的问题,提出 GradSamp,通过高斯分布采样参数更新并跳过部分 epoch,在多种视觉任务及 DA、DG、FL 场景中显著节能且保持性能。

摘要:

在深度学习(DL)模型训练期间,通过反向传播计算损失梯度会消耗大量能源。本文提出一种高效计算深度学习模型梯度的新方法,以缓解反向传播带来的巨大能源开销。利用深度学习模型的过参数化特性及其损失景观的平滑性,我们提出一种名为 GradSamp 的方法,用于从高斯分布中采样梯度更新。具体而言,我们在给定 epoch(周期性或随机选择)更新模型参数,方法是利用高斯“噪声”对来自前一 epoch 的参数进行逐元素扰动。该高斯分布的参数使用前两个 epoch 的模型参数值之间的误差来估计。GradSamp 不仅简化了梯度计算,还能够跳过整个 epoch,从而提升整体效率。我们在一系列多样的标准和非标准 CNN 及基于 transformer 的模型上严格验证了假设,涵盖图像分类、目标检测和图像分割等多种计算机视觉任务。此外,我们还探究了其在分布外场景中的有效性,例如域适应(DA)、域泛化(DG),以及联邦学习(FL)等去中心化设置。实验结果证实,GradSamp 能够在不损害性能的情况下实现显著的能源节约,凸显了其在实际深度学习应用中的多功能性和潜在影响。

英文摘要:

Computing the loss gradient via backpropagation consumes considerable energy during deep learning (DL) model training. In this paper, we propose a novel approach to efficiently compute DL models' gradients to mitigate the substantial energy overhead associated with backpropagation. Exploiting the over-parameterized nature of DL models and the smoothness of their loss landscapes, we propose a method called {\em GradSamp} for sampling gradient updates from a Gaussian distribution. Specifically, we update model parameters at a given epoch (chosen periodically or randomly) by perturbing the parameters (element-wise) from the previous epoch with Gaussian ``noise''. The parameters of the Gaussian distribution are estimated using the error between the model parameter values from the two previous epochs. {\em GradSamp} not only streamlines gradient computation but also enables skipping entire epochs, thereby enhancing overall efficiency. We rigorously validate our hypothesis across a diverse set of standard and non-standard CNN and transformer-based models, spanning various computer vision tasks such as image classification, object detection, and image segmentation. Additionally, we explore its efficacy in out-of-distribution scenarios such as Domain Adaptation (DA), Domain Generalization (DG), and decentralized settings like Federated Learning (FL). Our experimental results affirm the effectiveness of {\em GradSamp} in achieving notable energy savings without compromising performance, underscoring its versatility and potential impact in practical DL applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑