神经梯度接近对数正态：改进的量化和稀疏训练
Neural gradients are near-lognormal: improved quantized and sparse training
摘要：
虽然训练可以通过减少在整个模型中传播神经梯度所需的时间来加速，但大多数以前的工作都集中在权重和激活的量化/修剪上。这些方法通常不适用于具有非常不同的统计特性的神经梯度。区别于权重和激活，我们发现神经梯度的分布近似对数正态分布。考虑到这一点，我们提出了两个封闭形式的分析方法，以减少神经梯度的计算和记忆负担。第一种方法优化了梯度的浮点格式和比例。第二种方法准确地设置稀疏阈值梯度修剪。每种方法都在ImageNet上实现了最先进的结果。据我们所知，这篇论文是第一篇（1）将梯度转换为6位浮点格式，或者（2）实现高达85%的梯度稀疏性-在每种情况下都不会降低精度。参考实现随附在本文中。
英文摘要：
While training can mostly be accelerated by reducing the time needed to propagate neural gradients back throughout the model, most previous works focus on the quantization/pruning of weights and activations. These methods are often not applicable to neural gradients, which have very different statistical properties. Distinguished from weights and activations, we find that the distribution of neural gradients is approximately lognormal. Considering this, we suggest two closed-form analytical methods to reduce the computational and memory burdens of neural gradients. The first method optimizes the floating-point format and scale of the gradients. The second method accurately sets sparsity thresholds for gradient pruning. Each method achieves state-of-the-art results on ImageNet. To the best of our knowledge, this paper is the first to (1) quantize the gradients to 6-bit floating-point formats, or (2) achieve up to 85% gradient sparsity -- in each case without accuracy degradation. Reference implementation accompanies the paper.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
