幻觉检测的度量集成
Metric Ensembles For Hallucination Detection
AI总结：
本研究针对抽象摘要中的幻觉问题，评估多种无监督一致性度量，并提出基于LLM的集成方法，在wiki_bio_gpt3_hallucination数据集上超越现有最优水平。
摘要：
抽象式文本摘要近期引起了越来越多的关注，部分原因是大型语言模型（LLMs）的普及。与抽象式摘要生成相关的最紧迫问题之一是减少“幻觉”，即未包含在被摘要文档中的信息，这些信息可能完全错误。由于这一需求，人们提出了大量用于估计与所摘要文本一致性的度量。我们特别研究了一组用于摘要一致性的无监督度量，并在wiki_bio_gpt3_hallucination数据集中测量了它们彼此之间以及与人类评估分数的相关性。然后，我们将这些评估与由这些度量的简单线性集成构成的模型进行比较。我们发现，基于LLM的方法在幻觉检测方面优于其他无监督度量。我们还发现，只要集成中的度量具有足够相似且不相关的错误率，集成方法可以进一步提高这些分数。最后，我们提出了一种基于LLM评估的集成方法，并证明该方法优于先前的最先进水平。
英文摘要：
Abstractive text summarization has garnered increased interest as of late, in part due to the proliferation of large language models (LLMs). One of the most pressing problems related to generation of abstractive summaries is the need to reduce "hallucinations," information that was not included in the document being summarized, and which may be wholly incorrect. Due to this need, a wide array of metrics estimating consistency with the text being summarized have been proposed. We examine in particular a suite of unsupervised metrics for summary consistency, and measure their correlations with each other and with human evaluation scores in the wiki_bio_gpt3_hallucination dataset. We then compare these evaluations to models made from a simple linear ensemble of these metrics. We find that LLM-based methods outperform other unsupervised metrics for hallucination detection. We also find that ensemble methods can improve these scores even further, provided that the metrics in the ensemble have sufficiently similar and uncorrelated error rates. Finally, we present an ensemble method for LLM-based evaluations that we show improves over this previous SOTA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
