DiffSampling：提升神经文本生成的多样性和准确性
DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generation
AI总结：
DiffSampling通过分析令牌概率分布，提升神经文本生成的多样性和准确性，实验表明其在质量上与现有方法相当。
摘要：
尽管语言模型的能力不断增长，它们仍然频繁地从训练数据中重复内容，生成重复的文本，并倾向于常见的语法模式和词汇。可能的原因是解码策略：最常见的策略要么只考虑最可能的标记，从而减少输出多样性，要么增加不常见标记的可能性，从而影响输出的准确性和正确性。在本文中，我们提出了一种新的解码方法DiffSampling，该方法利用对令牌概率分布的数学分析，以确保生成上下文合适的文本。特别是，连续排序概率之间的差异可用于截断不正确的标记。此外，我们还提出了两种变体，旨在纠正常见采样策略的细微不一致之处。涉及四个不同文本生成任务的实验表明，尽管从更大的令牌集合中采样，我们的方法在质量上至少与现有方法相当。
英文摘要：
Despite their growing capabilities, language models still frequently reproduce content from their training data, generate repetitive text, and favor common grammatical patterns and vocabulary. A possible cause is the decoding strategy: the most common strategies either consider only the most probable tokens, which reduces output diversity, or increase the likelihood of unlikely tokens, compromising output accuracy and correctness. In this paper, we propose DiffSampling, a new decoding method that leverages a mathematical analysis of the token probability distribution to ensure the generation of contextually appropriate text. In particular, the difference between consecutive, sorted probabilities can be used to truncate incorrect tokens. In addition, we also propose two variations of the proposed method that aim to correct the subtle inconsistencies of common sampling strategies. Experiments involving four different text-generation tasks demonstrate that our approach consistently performs at least on par with the existing methods it builds upon in terms of quality, despite sampling from a larger set of tokens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
