arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.15090cs.CL
收藏

使用相似度评估摘要的事实一致性

Using Similarity to Evaluate Factual Consistency in Summaries

Intelligent Systems Laboratory(智能系统实验室);University of Bristol(布里斯托尔大学)

Yuxuan Ye, Edwin Simpson, Raul Santos Rodriguez

AI总结:

本文提出零样本事实性评估指标 SBERTScore,通过比较摘要与源文档的句子,无需微调即可媲美基于 NLI 和 QA 的指标,并证明结合多种技术能更有效检测摘要中的事实性错误。

摘要:

前沿的生成式摘要器能生成流畅的摘要,但无法保证生成文本的事实性。早期的摘要事实性评估指标通常基于 n-gram 重叠和嵌入相似度,但据报道无法与人类标注对齐。因此,许多检测事实不一致性的技术在自然语言推理(NLI)或问答(QA)模型基础上构建流水线,并增加额外的监督学习步骤。在本文中,我们重新审视基于相似度的指标,表明这种失败源于比较文本的选择及其粒度。我们提出了一种新的零样本事实性评估指标 Sentence-BERT Score(SBERTScore),该指标比较摘要与源文档之间的句子。它优于包括 BERTScore 在内的广泛使用的词-词指标,并且无需任何微调,就能在基准上与现有的基于 NLI 和 QA 的事实性指标相竞争。我们的实验表明,每种技术都有不同的优势,其中 SBERTScore 在识别正确摘要方面尤为有效。我们展示了如何组合多种技术以更有效地检测各类错误。

英文摘要:

Cutting-edge abstractive summarisers generate fluent summaries, but the factuality of the generated text is not guaranteed. Early summary factuality evaluation metrics are usually based on n-gram overlap and embedding similarity, but are reported fail to align with human annotations. Therefore, many techniques for detecting factual inconsistencies build pipelines around natural language inference (NLI) or question-answering (QA) models with additional supervised learning steps. In this paper, we revisit similarity-based metrics, showing that this failure stems from the comparison text selection and its granularity. We propose a new zero-shot factuality evaluation metric, Sentence-BERT Score (SBERTScore), which compares sentences between the summary and the source document. It outperforms widely-used word-word metrics including BERTScore and can compete with existing NLI and QA-based factuality metrics on the benchmark without needing any fine-tuning. Our experiments indicate that each technique has different strengths, with SBERTScore particularly effective in identifying correct summaries. We demonstrate how a combination of techniques is more effective in detecting various types of error.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑