arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.17042cs.LGcs.AIcs.CVcs.NE
收藏

用于增强神经网络训练的随机梯度采样

Stochastic Gradient Sampling for Enhancing Neural Networks Training

Stony Brook University(石溪大学)

Juyoung Yun

AI总结:

提出扩展 Adam 的 StochGradAdam 优化器,通过训练中选择性采样梯度子集降低计算成本,并在图像分类与分割任务中保持相当或更优性能。

摘要:

本文提出 StochGradAdam,这是一种作为 Adam 算法扩展而设计的新型优化器,融合随机梯度采样技术,以在保持稳健性能的同时提高计算效率。StochGradAdam 通过在训练期间选择性地采样一个梯度子集进行优化,从而降低计算成本,同时保留 Adam 所具有的自适应学习率和偏差校正优势。我们在图像分类和分割任务上的实验结果表明,即使每次迭代使用更少的梯度更新,StochGradAdam 也能取得与 Adam 相当甚至更优的性能。通过聚焦于关键梯度更新,StochGradAdam 可提供稳定收敛并增强对损失景观的探索,同时减轻噪声梯度的影响。结果表明,该方法对大规模模型和数据集尤其有效,为深度学习应用中的传统优化技术提供了一种有前景的替代方案。

英文摘要:

In this paper, we introduce StochGradAdam, a novel optimizer designed as an extension of the Adam algorithm, incorporating stochastic gradient sampling techniques to improve computational efficiency while maintaining robust performance. StochGradAdam optimizes by selectively sampling a subset of gradients during training, reducing the computational cost while preserving the advantages of adaptive learning rates and bias corrections found in Adam. Our experimental results, applied to image classification and segmentation tasks, demonstrate that StochGradAdam can achieve comparable or superior performance to Adam, even when using fewer gradient updates per iteration. By focusing on key gradient updates, StochGradAdam offers stable convergence and enhanced exploration of the loss landscape, while mitigating the impact of noisy gradients. The results suggest that this approach is particularly effective for large-scale models and datasets, providing a promising alternative to traditional optimization techniques for deep learning applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑