破解神经文本生成退化的奇特谜题
Closing the Curious Case of Neural Text Degeneration
AI总结：
该研究揭示了截断采样有效的理论原因，提出利用softmax瓶颈无需阈值判定词元真实概率的方法，开发的实验性截断策略在低熵开放式文本生成中表现优于阈值类方法。
摘要：
尽管截断采样启发式方法（如nucleus sampling核采样）在语言生成中无处不在，但其效果显著的原因仍不明确。我们通过证明以下结论为截断采样的有效性提供了理论解释：丢弃概率低于某一阈值的词元（最常见的截断类型）的截断方法，能够保证所有采样得到的词元都具有非零真实概率。然而，阈值是一种粗糙的启发式手段，必然也会丢弃一些具有非零真实概率的词元。为了寻求更精确的采样策略，我们证明可以利用已知的模型误差来源——softmax bottleneck（softmax瓶颈），无需依赖阈值就能判定某些词元具有非零真实概率。基于这些发现，我们开发了一种实验性截断策略，并开展了初步研究，证明了这类算法的应用前景。评估结果显示，在低熵（即接近贪心策略）的开放式文本生成任务中，我们的方法在自动评估指标和人工评估指标上均优于基于阈值的同类方法。我们的理论发现和初步实验不仅揭示了截断采样奏效的原因，也朝着开发更具表现力、能更好展现大语言模型生成能力的采样算法迈出了一步。
英文摘要：
Despite their ubiquity in language generation, it remains unknown why truncation sampling heuristics like nucleus sampling are so effective. We provide a theoretical explanation for the effectiveness of the truncation sampling by proving that truncation methods that discard tokens below some probability threshold (the most common type of truncation) can guarantee that all sampled tokens have nonzero true probability. However, thresholds are a coarse heuristic, and necessarily discard some tokens with nonzero true probability as well. In pursuit of a more precise sampling strategy, we show that we can leverage a known source of model errors, the softmax bottleneck, to prove that certain tokens have nonzero true probability, without relying on a threshold. Based on our findings, we develop an experimental truncation strategy and the present pilot studies demonstrating the promise of this type of algorithm. Our evaluations show that our method outperforms its threshold-based counterparts under automatic and human evaluation metrics for low-entropy (i.e., close to greedy) open-ended text generation. Our theoretical findings and pilot experiments provide both insight into why truncation sampling works, and make progress toward more expressive sampling algorithms that better surface the generative capabilities of large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
