重新审视类似 Pointer Networks 的架构以高效改进下一个词分布、摘要事实性及更多方面
Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond
AI总结：
本研究发现 Pointer Networks 能高效打破语言模型中的 softmax 瓶颈，据此提出多种 softmax 替代方案，在提升下一个词预测分布与摘要事实性的同时保持了计算效率。
摘要：
大多数语言模型（LMs）采用的输出 softmax 层，是否总是计算下一个词概率的最佳方式？鉴于现代基于 transformer 的 LM 中有如此多的注意力层，如今 Pointer Networks 是否已经多余？在本研究中，我们发现这两个问题的答案都是否定的。这是因为 softmax 瓶颈有时会阻碍 LM 预测出理想的分布，而 Pointer Networks 可以用来高效地打破这一瓶颈。基于这一发现，我们通过简化 Pointer Networks 和加速逐词重排序器，提出了几种 softmax 替代方案。在 GPT-2 中，我们的方案比最先进的 softmax 替代方案 mixture of softmax 效果显著更好且更高效。在摘要实验中，在不显著降低其训练/测试速度的情况下，我们基于 T5-Small 的最佳方法在 CNN/DM 和 XSUM 数据集上将 factCC 得分提高了 2 分，并在 BookSum 段落级数据集中将 MAUVE 得分提高了 30%。
英文摘要：
Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability? Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays? In this study, we discover that the answers to both questions are no. This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently. Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-by-word rerankers. In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative. In summarization experiments, without significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30% in BookSum paragraph-level dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
