Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)
Steinbuch Centre for Computing (SCC)（施泰因布赫计算中心（SCC））；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT））；Institute for Software Technology (SC)（软件技术研究所（SC））；German Aerospace Center (DLF)（德国航空航天中心（DLF））
Daniel Coquelin, Charlotte Debus, Markus Götz, Fabrice von der Lehr, James Kahn, Martin Siggel, Achim Streit
With increasing data and model complexities, the time required to train neural networks has become prohibitively large. To address the exponential rise in training time, users are turning to data parallel neural networks (DPNN) to utilize large-scale distributed resources on computer clusters. Current DPNN approaches implement the network parameter updates by synchronizing and averaging gradients across all processes with blocking communication operations. This synchronization is the central algorithmic bottleneck. To combat this, we introduce the Distributed Asynchronous and Selective Optimization (DASO) method which leverages multi-GPU compute node architectures to accelerate network training. DASO uses a hierarchical and asynchronous communication scheme comprised of node-local and global networks while adjusting the global synchronization rate during the learning process. We show that DASO yields a reduction in training time of up to 34% on classical and state-of-the-art networks, as compared to other existing data parallel training methods.
Efficient Asynchronous Federated Learning with Sparsification and Quantization
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Hithink RoyalFlush Information Network Co., Ltd.（同花顺信息服务股份有限公司）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术及产业化协同创新中心）；Department of Sciences and Informatics, Muroran Institute of Technology（室兰工业大学科学与信息学系）；Boston Consulting Group（波士顿咨询公司）
Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics
DeepCybo；Southern University of Science and Technology（南方科技大学）；Hong Kong Polytechnic University（香港理工大学）；Zhongguancun Academy（中关村学院）；Nanyang Technological University（南洋理工大学）
Catch-Up Distillation: You Only Need to Train Once for Accelerating Sampling
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Southeast University（东南大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）
Accelerating scientific discovery with Co-Scientist
Google Cloud AI Research（谷歌云人工智能研究）；Google DeepMind（谷歌DeepMind）；Google Research（谷歌研究）；Stanford University School of Medicine（斯坦福大学医学院）；Houston Methodist（休斯顿卫理公会医院）；Sequome；Fleming Initiative and Imperial College London（弗莱明倡议与伦敦帝国理工学院）
PAL -- Parallel active learning for machine-learned potentials
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Institute of Theoretical Informatics（理论信息学研究所）；Institute of Nanotechnology（纳米技术研究所）；Institute for Physical Chemistry（物理化学研究所）
