Optimizer Fusion: Efficient Training with Better Locality and Parallelism
The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Zixuan Jiang, Jiaqi Gu, Mingjie Liu, Keren Zhu, David Z. Pan
备注：
It is published as a paper at the Hardware Aware Efficient Training (HAET) workshop of ICLR 2021. There are 4 pages excluding references and appendices
Machine learning frameworks adopt iterative optimizers to train neural networks. Conventional eager execution separates the updating of trainable parameters from forward and backward computations. However, this approach introduces nontrivial training time overhead due to the lack of data locality and computation parallelism. In this work, we propose to fuse the optimizer with forward or backward computation to better leverage locality and parallelism during training. By reordering the forward computation, gradient calculation, and parameter updating, our proposed method improves the efficiency of iterative optimizers. Experimental results demonstrate that we can achieve an up to 20% training time reduction on various configurations. Since our methods do not alter the optimizer algorithm, they can be used as a general "plug-in" technique to the training process.
Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator
School of Computing, Institute of Science Tokyo（科学技术学院计算机学院）；Turing. inc（图灵公司）；Super Computing Research Center, Institute of Science Tokyo（科学技术学院超级计算研究中心）
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）；The University of Chicago Booth School of Business（芝加哥大学布斯商学院）；Northwestern University（西北大学）；University of Southern California Marshall School of Business（南加州大学马歇尔商学院）
PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction
College of Computer Science and Technology, Taiyuan University of Technology（太原理工大学计算机科学与技术学院）；School of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Computer and Information Technology, Shanxi University（山西大学计算机与信息技术学院）
Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies
Division of Applied Mathematics, Brown University（布朗大学应用数学系）；Euler Institute, Università della Svizzera italiana（瑞士意大利语区大学欧拉研究所）；UniDistance, Switzerland（瑞士远程大学）
