Training a sparse neural network from scratch requires optimizing connections at the same time as the weights themselves. Typically, the weights are redistributed after a predefined number of weight updates, removing a fraction of the parameters of each layer and inserting them at different locations in the same layers. The density of each layer is determined using heuristics, often purely based on the size of the parameter tensor. While the connections per layer are optimized multiple times during training, the density of each layer remains constant. This leaves great unrealized potential, especially in scenarios with a high sparsity of 90% and more. We propose Global Gradient-based Redistribution, a technique which distributes weights across all layers - adding more weights to the layers that need them most. Our evaluation shows that our approach is less prone to unbalanced weight distribution at initialization than previous work and that it is able to find better performing sparse subnetworks at very high sparsity levels.
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Minnesota Twin Cities(明尼苏达大学双城分校);North Carolina State University(北卡罗来纳州立大学);Meta AI;Stevens Institute of Technology(史蒂文斯理工学院)
Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off
University of Connecticut(康涅狄格大学);Texas A&M University(得克萨斯农工大学);North Carolina State University(北卡罗来纳州立大学);Lehigh University(理海大学);University of Texas at San Antonio(得克萨斯大学圣安东尼奥分校)
Dual-Balancing for Physics-Informed Neural Networks
Department of Computer Science, Hong Kong Baptist University(香港 Baptist 大学计算机科学系);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(高性能计算研究所(IHPC),科技研究局(A*STAR))
AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Shenzhen University of Advanced Technology(深圳大学);University of Oxford(牛津大学);University of Surrey(萨里大学)