接下来是什么？评估神经文本生成器面对人类产出变异性时的不确定性
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
AI总结：
本文研究神经文本生成器在NLG任务中面对人类产出变异时的不确定性，通过实例级方法分析模型概率分布与解码算法，证明多采样和多参考探查能更好理解模型的不确定性表示。
摘要：
在自然语言生成（NLG）任务中，对于任何输入，多个交流目标都是合理的，且任何目标都可以用多种方式表述或产出。我们刻画了在四个 NLG 任务中，人类产出在词汇、句法和语义层面的变化程度，并将人类产出变异与偶然性或数据不确定性联系起来。接着，我们检查由生成系统的预测概率分布和解码算法所塑造的输出字符串空间，以探究其不确定性。对于每个测试输入，我们测量生成器对人类产出变异的校准情况。遵循这种实例级方法，我们分析了 NLG 模型与解码策略，表明通过多采样及（在可能时）多参考探查生成器，能提供理解模型不确定性表示所需的细节水平。代码见 https://github.com/dmg-illc/nlg-uncertainty-probes。
英文摘要：
In Natural Language Generation (NLG) tasks, for any input, multiple communicative goals are plausible, and any goal can be put into words, or produced, in multiple ways. We characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. We then inspect the space of output strings shaped by a generation system's predicted probability distribution and decoding algorithm to probe its uncertainty. For each test input, we measure the generator's calibration to human production variability. Following this instance-level approach, we analyse NLG models and decoding strategies, demonstrating that probing a generator with multiple samples and, when possible, multiple references, provides the level of detail necessary to gain understanding of a model's representation of uncertainty. Code available at https://github.com/dmg-illc/nlg-uncertainty-probes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
