arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.11009cs.CL
收藏

开放文本生成中束搜索的标签偏差研究

Investigating Label Bias in Beam Search for Open-ended Text Generation

Yuanfudao AI Lab(猿辅导AI实验室)

Liang Wang, Jinlong Liu, Jingming Liu

摘要:

束搜索是一种有效的解码算法,在许多序列到序列(seq2seq)文本生成任务中得到广泛应用。然而,在开放式文本生成中,束搜索经常被发现产生重复和通用的文本,基于采样的解码算法,如top-k采样和核采样是更优选的。标准seq2seq模型由于其局部归一化概率公式而遭受标签偏差。本文提供了一系列的经验证据,标签偏见是一个主要的原因,这种退化行为的束搜索。通过结合局部归一化最大似然估计和全局归一化序列级训练,可以在几乎不牺牲困惑度的情况下降低标签偏差。为了定量测量标签偏差,我们测试了模型区分地面实况文本和一组上下文无关干扰项的能力。我们在大规模响应生成数据集上进行实验。结果表明,束搜索可以产生更多样化和有意义的文本与我们的方法,在自动和人类的评价指标。我们的分析还提出了几个未来的工作方向对开放式文本生成的巨大挑战。

英文摘要:

Beam search is an effective and widely used decoding algorithm in many sequence-to-sequence (seq2seq) text generation tasks. However, in open-ended text generation, beam search is often found to produce repetitive and generic texts, sampling-based decoding algorithms like top-k sampling and nucleus sampling are more preferred. Standard seq2seq models suffer from label bias due to its locally normalized probability formulation. This paper provides a series of empirical evidence that label bias is a major reason for such degenerate behaviors of beam search. By combining locally normalized maximum likelihood estimation and globally normalized sequence-level training, label bias can be reduced with almost no sacrifice in perplexity. To quantitatively measure label bias, we test the model's ability to discriminate the groundtruth text and a set of context-agnostic distractors. We conduct experiments on large-scale response generation datasets. Results show that beam search can produce more diverse and meaningful texts with our approach, in terms of both automatic and human evaluation metrics. Our analysis also suggests several future working directions towards the grand challenge of open-ended text generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑