Deep learning models are dominating almost all artificial intelligence tasks such as vision, text, and speech processing. Stochastic Gradient Descent (SGD) is the main tool for training such models, where the computations are usually performed in single-precision floating-point number format. The convergence of single-precision SGD is normally aligned with the theoretical results of real numbers since they exhibit negligible error. However, the numerical error increases when the computations are performed in low-precision number formats. This provides compelling reasons to study the SGD convergence adapted for low-precision computations. We present both deterministic and stochastic analysis of the SGD algorithm, obtaining bounds that show the effect of number format. Such bounds can provide guidelines as to how SGD convergence is affected when constraints render the possibility of performing high-precision computations remote.
On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computation
Eindhoven University of Technology(埃因霍温理工大学);Università di Pisa(比萨大学)
School of Mathematics and Statistics, Wuhan University(武汉大学数学与统计学院);Hubei Key Laboratory of Computational Science, Wuhan University(武汉大学湖北计算科学重点实验室);School of Mathematical Sciences, Peking University(北京大学数学科学学院)
Efficient and Accurate Optimal Transport with Mirror Descent and Conjugate Gradients
University of Toronto, Department of Computer Science(多伦多大学计算机科学系);Vector Institute(向量研究所);Polytechnique Montréal(蒙特利尔大学);Mila - Quebec AI Institute(魁北克AI研究所)
本文提出 MDOT 方法,将温度退火与镜像下降统一,并用 GPU 并行非线性共轭梯度算法 PNCG 高效求解离散最优传输问题,在弱正则化条件下以更高精度和更快速度超越现有 OT 求解器。
High-dimension Tensor Completion via Gradient-based Optimization Under Tensor-train Format
Graduate School of Engineering, Saitama Institute of Technology, Japan(日本埼玉科技大学工学研究科);Tensor Learning Unit, RIKEN Center for Advanced Intelligence Project (AIP), Japan(日本RIKEN先进人工智能项目(AIP)张量学习单元);School of Automation, Guangdong University of Technology, China(广东技术大学自动化学院);School of Computer Science and Technology, Hangzhou Dianzi University, China(杭州电子科技大学计算机科学与技术学院)
On the Power of Differentiable Learning versus PAC and SQ Learning
École polytechnique fédérale de Lausanne (EPFL)(洛桑联邦理工学院);Google Research(谷歌研究院);Hebrew University of Jerusalem (HUJI)(耶路撒冷希伯来大学);Massachusetts Institute of Technology (MIT)(麻省理工学院);Toyota Technological Institute at Chicago (TTIC)(芝加哥丰田理工学院)
Grokking phase transitions in learning local rules with gradient descent
Faculty of computer and information science, University of Ljubljana(卢布尔雅那大学计算机与信息科学学院);Faculty of mathematics and physics, University of Ljubljana(卢布尔雅那大学数学与物理学院)
Accelerating Generalized Linear Models with MLWeaving: A One-Size-Fits-All System for Any-precision Learning (Technical Report)
Systems Group, Department of Computer Science ETH Zurich(苏黎世联邦理工学院计算机科学系系统组);ETH Zurich(苏黎世联邦理工学院);Department of Computer Science ETH Zurich(苏黎世联邦理工学院计算机科学系);Computer Science ETH Zurich(苏黎世联邦理工学院计算机科学)
Meta-learning Spiking Neural Networks with Surrogate Gradient Descent
Peter Grünberg Institute – Neuromorphic Software Ecosystems(彼得·格伦贝格研究所——神经形态软件生态系统);Forschungszentrum Jülich(于利希研究中心);RWTH Aachen(亚琛工业大学);UC Irvine(加州大学欧文分校)
Bilevel Learning via Strategically Inexact Stochastic Gradient Descent
EPSRC Centre for Doctoral Training in Statistical Applied Mathematics at Bath(巴斯大学统计应用数学博士培训中心);Australian Research Council(澳大利亚研究理事会);Indian Institute of Technology (IIT), Kharagpur(印度理工学院(Kharagpur分校));University of Melbourne(墨尔本大学);Department of Electronics & Electrical Communication Engineering(电子与电气通信工程系);School of Mathematics and Statistics(数学与统计学学院)
On the role of synaptic stochasticity in training low-precision neural networks
Bocconi Institute for Data Science and Analytics, Bocconi University(博科尼大学博科尼数据科学与分析研究所);Italian Institute for Genomic Medicine(意大利基因组医学研究所);Istituto Nazionale di Fisica Nucleare(意大利国家核物理研究所);Politecnico di Torino(都灵理工大学);Radboud University Nijmegen(内梅亨大学);Donders Institute for Brain, Cognition and Behaviour(唐德斯大脑、认知与行为研究所);International Centre for Theoretical Physics(国际理论物理中心)
Isomorphic Functionalities between Ant Colony and Ensemble Learning: Part III -- Gradient Descent, Neural Plasticity, and the Emergence of Deep Intelligence
School of Mathematics and Statistics, Rochester Institute of Technology(罗彻斯特理工学院数学与统计学院);Gosnell School of Life Sciences, Rochester Institute of Technology(罗彻斯特理工学院戈斯内尔生命科学学院);Center for Imaging Science, Rochester Institute of Technology(罗彻斯特理工学院成像科学中心)
Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantization
College of Software, Northeastern University(东北大学软件学院);Institute of Intelligent Manufacturing, Nanjing Tech University(南京科技大学智能制造研究院);Faculty of Engineering, University of Toyama(东大大学工学院)
QB Information Technologies (1QBit)(1QB信息技术公司(1QBit));University of British Columbia(不列颠哥伦比亚大学);Institute for Quantum Computing(量子计算研究所);University of Waterloo(滑铁卢大学)
eGAD! double descent is explained by Generalized Aliasing Decomposition
Department of Physics and Astronomy, Brigham Young University, Provo, Utah 84602, USA(物理与天文学系, BYU,犹他州普罗沃,84602,USA);Department of Mathematics, Brigham Young University, Provo, Utah 84602, USA(数学系, BYU,犹他州普罗沃,84602,USA)
AMD(超威半导体公司);Fermi National Accelerator Laboratory(费米国家加速器实验室);Massachusetts Institute of Technology(麻省理工学院);European Organization for Nuclear Research (CERN)(欧洲核子研究组织(CERN));Heidelberg University(海德堡大学);Harvard University(哈佛大学);University of Washington(华盛顿大学)
Characterization and Mitigation of Training Instabilities in Microscaling Formats
Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(自然与人工智能研究学院,哈佛大学);Department of Computer Science, Harvard University(计算机科学系,哈佛大学)