arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.00666cs.CLcs.AI
收藏

局部典型采样

Locally Typical Sampling

ETH Zürich(苏黎世联邦理工学院);University of Cambridge(剑桥大学)

Clara Meister, Tiago Pimentel, Gian Wiher, Ryan Cotterell

AI总结:

本文提出局部典型采样方法,通过约束生成词的信息内容接近模型条件熵,在保持文本质量的同时减少重复退化,并在摘要和故事生成任务中取得与核采样及 top-k 采样相当的性能。

摘要:

当今的概率语言生成器在生成连贯流畅的文本方面表现不佳,尽管其底层模型在标准指标(如困惑度)下表现良好。这一差异在过去几年中一直困扰着语言生成领域。在这项工作中,我们提出,将自然语言生成抽象为离散随机过程——从而允许进行信息论分析——可以为概率语言生成器的行为提供新的见解,例如为什么高概率文本可能显得枯燥或重复。人类使用语言作为传递信息的手段,旨在同时以高效和最小化错误的方式进行;事实上,心理语言学研究表表明,人类在选择字符串中的每个词时都带有这种潜意识目标。我们正式定义了满足这一标准的字符串集合:即每个词的信息内容接近期望信息内容,也就是我们模型的条件熵。随后,我们提出了一种简单而高效的程序,用于在从概率模型生成时强制执行这一标准,我们称之为局部典型采样。自动评估和人工评估表明,与核采样和 top-k 采样相比,局部典型采样在质量方面提供了具有竞争力的性能(在抽象式摘要和故事生成中均如此),同时持续减少了退化性重复。

英文摘要:

Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics, e.g., perplexity. This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language generation as a discrete stochastic process--which allows for an information-theoretic analysis--can provide new insights into the behavior of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We formally define the set of strings that meet this criterion: those for which each word has an information content close to the expected information content, i.e., the conditional entropy of our model. We then propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑