arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.06784cs.CL
收藏

稳定熵假设与熵感知解码:鲁棒自然语言生成分析与算法

The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

Mila(米拉研究所);McGill University(麦吉尔大学);DeepMind(深度思维);Meta AI

Kushal Arora, Timothy J. O'Donnell, Doina Precup, Jason Weston, Jackie C. K. Cheung

摘要:

最先进的语言生成模型在应用于开放式生成问题(如文本补全、故事生成或对话建模)时可能会退化。这种退化通常表现为不连贯、缺乏词汇多样性、自我重复或从上下文中复制。在本文中,我们假设“类人”代通常位于一个狭窄的和几乎平坦的熵带,这些熵边界的违反与退化行为。我们的实验表明,这种稳定的窄熵区存在于模型,任务和域中,并证实了这一假设,即违反这一区域与退化相关。然后,我们使用这种洞察力,提出了一个熵感知解码算法,尊重这些熵的界限,从而减少退化,更多的上下文,和“类人”的语言生成在开放式文本生成设置。

英文摘要:

State-of-the-art language generation models can degenerate when applied to open-ended generation problems such as text completion, story generation, or dialog modeling. This degeneration usually shows up in the form of incoherence, lack of vocabulary diversity, and self-repetition or copying from the context. In this paper, we postulate that ``human-like'' generations usually lie in a narrow and nearly flat entropy band, and violation of these entropy bounds correlates with degenerate behavior. Our experiments show that this stable narrow entropy zone exists across models, tasks, and domains and confirm the hypothesis that violations of this zone correlate with degeneration. We then use this insight to propose an entropy-aware decoding algorithm that respects these entropy bounds resulting in less degenerate, more contextual, and "human-like" language generation in open-ended text generation settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑