Despite impressive performance, deep neural networks require significant memory and computation costs, prohibiting their application in resource-constrained scenarios. Sparse training is one of the most common techniques to reduce these costs, however, the sparsity constraints add difficulty to the optimization, resulting in an increase in training time and instability. In this work, we aim to overcome this problem and achieve space-time co-efficiency. To accelerate and stabilize the convergence of sparse training, we analyze the gradient changes and develop an adaptive gradient correction method. Specifically, we approximate the correlation between the current and previous gradients, which is used to balance the two gradients to obtain a corrected gradient. Our method can be used with the most popular sparse training pipelines under both standard and adversarial setups. Theoretically, we prove that our method can accelerate the convergence rate of sparse training. Extensive experiments on multiple datasets, model architectures, and sparsities demonstrate that our method outperforms leading sparse training methods by up to \textbf{5.0\%} in accuracy given the same number of training epochs, and reduces the number of training epochs by up to \textbf{52.1\%} to achieve the same accuracy. Our code is available on: \url{https://github.com/StevenBoys/AGENT}.
Confidence Adjusted Surprise Measure for Active Resourceful Trials (CA-SMART): A Data-driven Active Learning Framework for Accelerating Material Discovery under Resource Constraints
West Virginia University（西弗吉尼亚大学）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Confidence Trigger Detection: Accelerating Real-time Tracking-by-detection Systems
Columbia University（哥伦比亚大学）；Cornell University（康奈尔大学）；Pace University（佩斯大学）；University of Michigan（密歇根大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；New York University（纽约大学）
RAPNet: Accelerating Algebraic Multigrid with Learned Sparse Corrections
Institute for Interdisciplinary Computational Sciences, Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Be'er Sheva, Israel（交叉学科计算科学研究所，计算机与信息科学学院，内盖夫本·古里安大学，以色列贝尔谢瓦）；Department of Mathematics and Computer Science, Emory University, Atlanta, GA, USA（数学与计算机科学系，埃默里大学，美国亚特兰大，GA）
Accelerating Neural ODEs Using Model Order Reduction
Tampere University（坦佩雷大学）；Faculty of Medicine and Health Technology, Tampere University（坦佩雷大学医学与健康技术学院）；Faculty of Information Technology and Communication Sciences, Tampere University（坦佩雷大学信息技术与传播科学学院）
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
School of Computer Science and Technology（计算机科学与技术学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)（数据科学与分析方向，香港科技大学（广州））；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（计算机科学与工程系，香港科技大学）
