arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.14012cs.LG
收藏

基于梯度的权重密度平衡用于鲁棒动态稀疏训练

Gradient-based Weight Density Balancing for Robust Dynamic Sparse Training

Graz University of Technology(格拉茨技术大学);Huawei Technologies(华为技术有限公司)

Mathias Parger, Alexander Ertl, Paul Eibensteiner, Joerg H. Mueller, Martin Winter, Markus Steinberger

摘要:

从头开始训练稀疏神经网络需要在优化权重本身的同时优化连接。通常,权重在预定义数量的权重更新之后被重新分布,移除每层的一部分参数并将它们插入相同层中的不同位置。每一层的密度都是使用几何学来确定的,通常纯粹基于参数张量的大小。虽然每层的连接在训练期间被优化多次,但每层的密度保持不变。这留下了巨大的未实现的潜力,特别是在具有90%或更高稀疏度的场景中。我们提出了基于全局一致性的重分布,这是一种在所有层中分布权重的技术-将更多权重添加到最需要它们的层。我们的评估表明,我们的方法是不太容易不平衡的权重分布在初始化比以前的工作,它是能够找到更好地执行稀疏子网在非常高的稀疏水平。

英文摘要:

Training a sparse neural network from scratch requires optimizing connections at the same time as the weights themselves. Typically, the weights are redistributed after a predefined number of weight updates, removing a fraction of the parameters of each layer and inserting them at different locations in the same layers. The density of each layer is determined using heuristics, often purely based on the size of the parameter tensor. While the connections per layer are optimized multiple times during training, the density of each layer remains constant. This leaves great unrealized potential, especially in scenarios with a high sparsity of 90% and more. We propose Global Gradient-based Redistribution, a technique which distributes weights across all layers - adding more weights to the layers that need them most. Our evaluation shows that our approach is less prone to unbalanced weight distribution at initialization than previous work and that it is able to find better performing sparse subnetworks at very high sparsity levels.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑