ChatGPT作为文本摘要的事实不一致性评估器
ChatGPT as a Factual Inconsistency Evaluator for Text Summarization
AI总结：
本文探索ChatGPT在零样本设置下评估文本摘要事实不一致性的能力，实验表明其在蕴含推理、摘要排序和一致性评分三项任务上普遍优于以往指标，但也存在偏好词汇相似候选、错误推理和理解指令不足等局限。
摘要：
预训练语言模型极大地提升了文本摘要的性能。现有方法的一个主要问题是，大多数生成的摘要在事实上与其源文档不一致。为了缓解这一问题，许多工作致力于开发基于自然语言推理、问答和句法依存等的有效事实性评估指标。然而，这些方法要么受限于较高的计算复杂度，要么受限于多组件流水线引入的不确定性，导致其与人类判断仅部分一致。最近，大语言模型（LLMs）不仅在文本生成方面表现出色，在语言理解方面也展现出优异性能。在本文中，我们特别探索了ChatGPT在零样本设置下评估事实不一致性的能力，通过在粗粒度和细粒度评估任务上进行检验，包括二元蕴含推理、摘要排序和一致性评分。实验结果表明，ChatGPT在这三项任务上普遍优于以往的评估指标，表明其在事实不一致性评估方面具有巨大潜力。然而，对ChatGPT输出的更仔细检查揭示了某些局限性，包括其偏好词汇上更相似的候选摘要、错误推理以及对指令理解不足。
英文摘要：
The performance of text summarization has been greatly boosted by pre-trained language models. A main concern of existing methods is that most generated summaries are not factually inconsistent with their source documents. To alleviate the problem, many efforts have focused on developing effective factuality evaluation metrics based on natural language inference, question answering, and syntactic dependency et al. However, these approaches are limited by either their high computational complexity or the uncertainty introduced by multi-component pipelines, resulting in only partial agreement with human judgement. Most recently, large language models(LLMs) have shown excellent performance in not only text generation but also language comprehension. In this paper, we particularly explore ChatGPT's ability to evaluate factual inconsistency under a zero-shot setting by examining it on both coarse-grained and fine-grained evaluation tasks including binary entailment inference, summary ranking, and consistency rating. Experimental results indicate that ChatGPT generally outperforms previous evaluation metrics across the three tasks, indicating its great potential for factual inconsistency evaluation. However, a closer inspection of ChatGPT's output reveals certain limitations including its preference for more lexically similar candidates, false reasoning, and inadequate understanding of instructions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
