arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.01651cs.LGcs.NAmath.NA
收藏

低精度数列中随机梯度下降的收敛性

On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats

Polytechnique Montreal(蒙特利尔综合理工学院);University of Pisa(比萨大学);McGill University(麦吉尔大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Matteo Cacciola, Antonio Frangioni, Masoud Asgharian, Alireza Ghaffari, Vahid Partovi Nia

摘要:

深度学习模型正在主导几乎所有的人工智能任务,如视觉、文本和语音处理。随机梯度下降(SGD)是训练此类模型的主要工具,其中计算通常以单精度浮点数格式执行。单精度SGD的收敛性通常与实数的理论结果一致,因为它们的误差可以忽略不计。然而,当以低精度数字格式执行计算时,数值误差增加。这为研究适用于低精度计算的SGD收敛性提供了令人信服的理由。我们提出了确定性和随机分析的SGD算法,获得的界限,显示的效果的数字格式。这样的界限可以提供指导方针,如何SGD收敛的影响时,约束,使远程执行高精度计算的可能性。

英文摘要:

Deep learning models are dominating almost all artificial intelligence tasks such as vision, text, and speech processing. Stochastic Gradient Descent (SGD) is the main tool for training such models, where the computations are usually performed in single-precision floating-point number format. The convergence of single-precision SGD is normally aligned with the theoretical results of real numbers since they exhibit negligible error. However, the numerical error increases when the computations are performed in low-precision number formats. This provides compelling reasons to study the SGD convergence adapted for low-precision computations. We present both deterministic and stochastic analysis of the SGD algorithm, obtaining bounds that show the effect of number format. Such bounds can provide guidelines as to how SGD convergence is affected when constraints render the possibility of performing high-precision computations remote.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑