GCWSNet：用于神经网络可扩展和精确训练的广义一致加权采样
GCWSNet: Generalized Consistent Weighted Sampling for Scalable and Accurate Training of Neural Networks
摘要：
我们开发了“广义一致加权采样”（GCWS）的散列“powered-GMM”（pGMM）内核（与调优参数$p$）。事实证明，GCWS提供了一个数值稳定的计划，应用电源转换的原始数据，无论大小的$p$和数据。功率变换通常对提高性能有效，在许多情况下相当有效。我们将散列数据馈送到各种公共分类数据集上的神经网络，并将我们的方法命名为“GCWSNet”。我们广泛的实验表明，GCWSNet往往提高分类精度。此外，从实验中可以明显看出，GCWSNet收敛得更快。事实上，GCWS通常只需要（少于）一个训练过程的时间就可以达到合理的精度。该属性是非常期望的，因为许多应用，诸如广告点击率（CTR）预测模型或数据流（即，数据只看到一次），通常只训练一个时期。另一个有益的副作用是，神经网络的第一层的计算变成了加法而不是乘法，因为输入数据变成了二进制（并且高度稀疏）。经验比较（归一化）随机傅立叶特征（NRFF）。本文还提出了通过计数草图来减小GCWSNet的模型规模，并为分析计数草图对GCWS精度的影响提供了理论依据。我们的分析表明，“8位”策略应该工作得很好，因为我们总是可以在GCWS散列的输出上应用8位计数草图散列，而不会对准确性造成太大影响。在训练深度神经网络时，还有许多其他方法可以利用GCWS。例如，可以在最后一层的输出上应用GCWS，以提高训练的深度神经网络的准确性。
英文摘要：
We develop the "generalized consistent weighted sampling" (GCWS) for hashing the "powered-GMM" (pGMM) kernel (with a tuning parameter $p$). It turns out that GCWS provides a numerically stable scheme for applying power transformation on the original data, regardless of the magnitude of $p$ and the data. The power transformation is often effective for boosting the performance, in many cases considerably so. We feed the hashed data to neural networks on a variety of public classification datasets and name our method ``GCWSNet''. Our extensive experiments show that GCWSNet often improves the classification accuracy. Furthermore, it is evident from the experiments that GCWSNet converges substantially faster. In fact, GCWS often reaches a reasonable accuracy with merely (less than) one epoch of the training process. This property is much desired because many applications, such as advertisement click-through rate (CTR) prediction models, or data streams (i.e., data seen only once), often train just one epoch. Another beneficial side effect is that the computations of the first layer of the neural networks become additions instead of multiplications because the input data become binary (and highly sparse). Empirical comparisons with (normalized) random Fourier features (NRFF) are provided. We also propose to reduce the model size of GCWSNet by count-sketch and develop the theory for analyzing the impact of using count-sketch on the accuracy of GCWS. Our analysis shows that an ``8-bit'' strategy should work well in that we can always apply an 8-bit count-sketch hashing on the output of GCWS hashing without hurting the accuracy much. There are many other ways to take advantage of GCWS when training deep neural networks. For example, one can apply GCWS on the outputs of the last layer to boost the accuracy of trained deep neural networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
