TofuEval：评估大语言模型在主题聚焦对话摘要上的幻觉
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization
AI总结：
本文提出主题聚焦对话摘要评估基准TofuEval，通过人工标注和错误分类分析揭示现有LLM在对话摘要中幻觉严重且作为评估器表现不佳，而非LLM指标更优。
摘要：
近年来，单文档新闻摘要的忠实性研究取得了实质性进展，这得益于对事实一致性或幻觉评估的研究。我们探究这些进展是否能够迁移到其他文本摘要领域。我们提出了一个新的主题聚焦对话摘要评估基准，该基准由不同规模的大语言模型（LLM）生成。我们提供了这些摘要事实一致性的句子级二元人工标注，以及对事实不一致句子的详细解释。我们的分析表明，现有的LLM在对话领域会产生大量事实错误，无论模型规模大小。另一方面，当LLM（包括GPT-4）作为二元事实评估器时，其表现不佳，且可能被当前最先进的专用事实性评估指标所超越。最后，我们基于一个精心整理的错误分类体系对幻觉类型进行了分析。我们发现，模型生成的摘要中存在多样的错误和错误分布，并且非LLM指标比基于LLM的评估器能够更好地捕获所有错误类型。
英文摘要：
Single document news summarization has seen substantial progress on faithfulness in recent years, driven by research on the evaluation of factual consistency, or hallucinations. We ask whether these advances carry over to other text summarization domains. We propose a new evaluation benchmark on topic-focused dialogue summarization, generated by LLMs of varying sizes. We provide binary sentence-level human annotations of the factual consistency of these summaries along with detailed explanations of factually inconsistent sentences. Our analysis shows that existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model's size. On the other hand, when LLMs, including GPT-4, serve as binary factual evaluators, they perform poorly and can be outperformed by prevailing state-of-the-art specialized factuality evaluation metrics. Finally, we conducted an analysis of hallucination types with a curated error taxonomy. We find that there are diverse errors and error distributions in model-generated summaries and that non-LLM based metrics can capture all error types better than LLM-based evaluators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
