arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.08436cs.CL
收藏

不要说你不知道的:通过约束束搜索提高抽象式摘要的一致性

Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search

Allen Institute for AI(艾伦人工智能研究所);Masakhane(马萨卡内);University of Washington(华盛顿大学);MosaicML;MIT(麻省理工学院);Northwestern University(西北大学)

Daniel King, Zejiang Shen, Nishant Subramani, Daniel S. Weld, Iz Beltagy, Doug Downey

AI总结:

本研究提出PINOCCHIO解码方法,通过约束束搜索检测并避免幻觉,将抽象式摘要的一致性平均提升约67%。

摘要:

当前的抽象式摘要系统能够生成流畅且相关的输出,但常常“幻觉”出源文本不支持的内容。我们分析了幻觉与训练数据之间的联系,并发现证据表明模型产生幻觉是因为它们训练时使用的目标摘要缺乏源文本的支持。基于我们的发现,我们提出了PINOCCHIO,一种新的解码方法,通过约束束搜索来避免幻觉,从而提高基于Transformer的抽象式摘要器的一致性。在给定模型状态和某一步骤的输出时,PINOCCHIO基于对源文本归因的各种度量来检测可能的模型幻觉。PINOCCHIO会回溯以寻找更一致的输出,并且在找不到一致生成时可以完全选择不生成摘要。在实验中,我们发现PINOCCHIO在两个抽象式摘要数据集上将生成的一致性(以F1衡量)平均提高了约67%。

英文摘要:

Abstractive summarization systems today produce fluent and relevant output, but often "hallucinate" statements not supported by the source text. We analyze the connection between hallucinations and training data, and find evidence that models hallucinate because they train on target summaries that are unsupported by the source. Based on our findings, we present PINOCCHIO, a new decoding method that improves the consistency of a transformer-based abstractive summarizer by constraining beam search to avoid hallucinations. Given the model states and outputs at a given step, PINOCCHIO detects likely model hallucinations based on various measures of attribution to the source text. PINOCCHIO backtracks to find more consistent output, and can opt to produce no summary at all when no consistent generation can be found. In experiments, we find that PINOCCHIO improves the consistency of generation (in terms of F1) by an average of~67% on two abstractive summarization datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑