论语言生成中的概率-质量悖论
On the probability-quality paradox in language generation
摘要:
当从神经概率模型生成自然语言时,高概率并不总是与高质量一致:经常观察到模式搜索解码方法,即,那些在模型下产生高概率文本的,导致不自然的语言。另一方面,由随机方法生成的低概率文本被认为更像人类。本文从信息论的角度对语言生成进行了分析,并对这一现象进行了解释。具体来说,我们认为类人语言应该包含一定量的信息(量化为负对数概率),接近自然字符串上分布的熵。此外,我们认为,语言与实质上更多(或更少)的信息是不可取的。我们提供了支持这一假设的初步经验证据;人类和机器生成的文本的质量评级-涵盖多个任务和常见的解码策略-表明高质量文本的信息内容比我们预期的更接近熵。
英文摘要:
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality: It has often been observed that mode-seeking decoding methods, i.e., those that produce high-probability text under the model, lead to unnatural language. On the other hand, the lower-probability text generated by stochastic methods is perceived as more human-like. In this note, we offer an explanation for this phenomenon by analyzing language generation through an information-theoretic lens. Specifically, we posit that human-like language should contain an amount of information (quantified as negative log-probability) that is close to the entropy of the distribution over natural strings. Further, we posit that language with substantially more (or less) information is undesirable. We provide preliminary empirical evidence in favor of this hypothesis; quality ratings of both human and machine-generated text -- covering multiple tasks and common decoding strategies -- suggest high-quality text has an information content significantly closer to the entropy than we would expect by chance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。