arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.15191cs.CL
收藏

截断抽样作为语言模型去平滑

Truncation Sampling as Language Model Desmoothing

Stanford University(斯坦福大学)

John Hewitt, Christopher D. Manning, Percy Liang

摘要:

来自神经语言模型的长文本样本的质量可能很差。截断抽样算法--如top- $p$或top- $k$ --通过在每一步将某些单词的概率设置为零来解决这个问题。这项工作提供了框架的截断的目的,并为该目的的改进算法。我们建议将神经语言模型视为真实分布和平滑分布的混合物,以避免无限的困惑。在这种情况下,截断算法的目的是执行去平滑,估计真实分布的支持度的子集。找到一个好的子集是至关重要的:我们表明,顶级$p$不必要地截断高概率的话,例如,导致它截断所有的话,但特朗普的文件开始与唐纳德。我们引入了$η$ -采样,它截断低于熵相关概率阈值的单词。与以前的算法相比,$η$ -sampling生成更合理的长英文文档根据人类,是更好地打破重复,并表现出更合理的电池上的测试分布。

英文摘要:

Long samples of text from neural language models can be of poor quality. Truncation sampling algorithms--like top-$p$ or top-$k$ -- address this by setting some words' probabilities to zero at each step. This work provides framing for the aim of truncation, and an improved algorithm for that aim. We propose thinking of a neural language model as a mixture of a true distribution and a smoothing distribution that avoids infinite perplexity. In this light, truncation algorithms aim to perform desmoothing, estimating a subset of the support of the true distribution. Finding a good subset is crucial: we show that top-$p$ unnecessarily truncates high-probability words, for example causing it to truncate all words but Trump for a document that starts with Donald. We introduce $η$-sampling, which truncates words below an entropy-dependent probability threshold. Compared to previous algorithms, $η$-sampling generates more plausible long English documents according to humans, is better at breaking out of repetition, and behaves more reasonably on a battery of test distributions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑