用大型语言模型评估摘要的事实一致性
Evaluating Factual Consistency of Summaries with Large Language Models
AI总结：
本研究通过直接提示大型语言模型来评估摘要的事实一致性，系统比较了多种模型和提示方法，实验证明该方法在多个设置下均优于现有最佳系统，准确率最高提升12.2个百分点。
摘要：
在摘要生成研究中，检测摘要中的事实错误一直是一个重要且具有挑战性的课题。受大型语言模型（LLMs）涌现能力的启发，我们探索通过直接提示LLMs来评估摘要的事实一致性。我们进行了一项全面的实证研究，以评估LLMs作为事实一致性评估器的能力，该研究包括：（1）分析不同的LLMs，如GPT模型系列和Flan-T5；（2）研究多种提示方法，包括普通提示、思维链提示以及一种逐句提示方法，以处理长摘要；（3）在由多种摘要系统生成的多样化摘要上进行评估，这些系统涵盖从预Transformer方法到最先进的预训练模型。我们的实验表明，在所有设置下，提示LLMs都能超越之前最佳的事实性系统，在不一致检测的二元分类准确率上最高提升12.2个绝对百分点。
英文摘要：
Detecting factual errors in summaries has been an important and challenging subject in summarization research. Inspired by the emergent ability of large language models (LLMs), we explore evaluating factual consistency of summaries by directly prompting LLMs. We present a comprehensive empirical study to assess the ability of LLMs as factual consistency evaluators, which consists of (1) analyzing different LLMs such as the GPT model series and Flan-T5; (2) investigating a variety of prompting methods including vanilla prompting, chain-of-thought prompting, and a sentence-by-sentence prompting method to tackle long summaries; and (3) evaluating on diverse summaries generated by multiple summarization systems, ranging from pre-transformer methods to SOTA pretrained models. Our experiments demonstrate that prompting LLMs is able to outperform the previous best factuality systems in all settings, by up to 12.2 absolute points in terms of the binary classification accuracy on inconsistency detection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
