通过可学习的单调逐点非线性打破Softmax瓶颈
Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities
摘要：
最后一个线性层上的Softmax函数是神经网络中输出概率分布的实际方法。在诸如语言模型或文本生成之类的许多应用中，该模型必须在大的输出词汇表上产生分布。最近，这已被证明具有有限的代表性的能力，由于其连接的秩瓶颈矩阵分解。然而，关于Linear-Softmax对于实际感兴趣的量（例如交叉熵或模式估计）的限制，我们知之甚少，这是我们在这里探索的一个方向。作为一个有效的解决方案，以减轻这个问题，我们建议学习参数单调函数的logits顶部。我们从理论上研究这种单调函数的秩增加能力。从经验上讲，我们的方法在合成和真实语言模型实验中在两个不同的质量指标上优于传统的Linear-Softmax层，几乎不增加时间或内存开销，同时与计算成本更高的Softmax混合物相当。
英文摘要：
The Softmax function on top of a final linear layer is the de facto method to output probability distributions in neural networks. In many applications such as language models or text generation, this model has to produce distributions over large output vocabularies. Recently, this has been shown to have limited representational capacity due to its connection with the rank bottleneck in matrix factorization. However, little is known about the limitations of Linear-Softmax for quantities of practical interest such as cross entropy or mode estimation, a direction that we explore here. As an efficient and effective solution to alleviate this issue, we propose to learn parametric monotonic functions on top of the logits. We theoretically investigate the rank increasing capabilities of such monotonic functions. Empirically, our method improves in two different quality metrics over the traditional Linear-Softmax layer in synthetic and real language model experiments, adding little time or memory overhead, while being comparable to the more computationally expensive mixture of Softmaxes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
