球形损失族的精确梯度更新与输出大小无关
Exact gradient updates in time independent of output size for the spherical loss family
摘要:
一类重要的问题涉及训练具有非常高维D的稀疏预测目标的深度神经网络。这些自然发生在例如神经语言模型或词嵌入的学习中,通常被设定为预测大小为D(例如200,000)的词汇表中的下一个词的概率。从合理维度d(例如,500)的最后隐藏层计算同样大但通常非稀疏的D维输出向量对于每个示例导致令人望而却步的O(Dd)计算成本,更新$D \times d$输出权重矩阵和计算反向传播到先前层所需的梯度也是如此。虽然有效处理大型稀疏网络输入是微不足道的,但大型稀疏目标的情况并非如此,因此迄今为止,在训练期间,诸如分层softmax或基于采样的近似之类的近似替代方案一直被回避。在这项工作中,我们开发了一个原始的算法方法,对于一个家庭的损失函数,包括平方误差和球形softmax,可以计算精确的损失,梯度更新的输出权重,和梯度的反向传播,所有在$O(d^{2})$每个例子,而不是$O(Dd)$,显着没有计算的D维输出。所提出的算法产生高达$D/4d$的加速比,即对于典型大小的两个数量级,对于在这种网络架构中通常支配训练时间的计算的关键部分。
英文摘要:
An important class of problems involves training deep neural networks with sparse prediction targets of very high dimension D. These occur naturally in e.g. neural language models or the learning of word-embeddings, often posed as predicting the probability of next words among a vocabulary of size D (e.g. 200,000). Computing the equally large, but typically non-sparse D-dimensional output vector from a last hidden layer of reasonable dimension d (e.g. 500) incurs a prohibitive O(Dd) computational cost for each example, as does updating the $D \times d$ output weight matrix and computing the gradient needed for backpropagation to previous layers. While efficient handling of large sparse network inputs is trivial, the case of large sparse targets is not, and has thus so far been sidestepped with approximate alternatives such as hierarchical softmax or sampling-based approximations during training. In this work we develop an original algorithmic approach which, for a family of loss functions that includes squared error and spherical softmax, can compute the exact loss, gradient update for the output weights, and gradient for backpropagation, all in $O(d^{2})$ per example instead of $O(Dd)$, remarkably without ever computing the D-dimensional output. The proposed algorithm yields a speedup of up to $D/4d$ i.e. two orders of magnitude for typical sizes, for that critical part of the computations that often dominates the training time in this kind of network architecture.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。