arXiv 2410.132102024-10-17cs.CLcs.AI

FaithBench：面向现代大语言模型摘要任务的多样化幻觉基准

FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs

Vectara, Inc.（Vectara公司）；Iowa State University（爱荷华州立大学）；University of Southern California（南加利福尼亚大学）；Entropy Technologies（熵技术公司）；University of Waterloo（滑铁卢大学）；University of Wisconsin, Madison（威斯康星大学麦迪逊分校）

Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Cheny… 展开作者

Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Chenyu Xu, Matthew Gonzales, Ofer Mendelevitch, Amin Ahmad

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出FaithBench基准，汇集10个现代LLM的挑战性摘要幻觉并由专家标注，发现GPT-4o和GPT-3.5-Turbo幻觉最少，但现有检测模型准确率仅约50%，揭示改进空间。

摘要：

摘要是大语言模型（LLMs）最常执行的任务之一，尤其是在检索增强生成（RAG）等应用中。然而，现有的对LLM生成摘要中幻觉的评估，以及对幻觉检测模型的评估，在考虑的LLM及其家族方面都缺乏多样性和时效性。本文介绍了FaithBench，一个摘要幻觉基准，包含来自8个不同家族的10个现代LLM所产生的具有挑战性的幻觉，并附有人类专家的真实标注。这里的“具有挑战性”指的是流行的、最先进的幻觉检测模型（包括作为评判者的GPT-4o）在这些摘要上存在分歧。我们的结果表明，GPT-4o和GPT-3.5-Turbo产生的幻觉最少。然而，即使是最好的幻觉检测模型在FaithBench上的准确率也接近50%，这表明未来还有很大的改进空间。代码仓库为https://github.com/vectara/FaithBench。

英文摘要：

Summarization is one of the most common tasks performed by large language models (LLMs), especially in applications like Retrieval-Augmented Generation (RAG). However, existing evaluations of hallucinations in LLM-generated summaries, and evaluations of hallucination detection models both suffer from a lack of diversity and recency in the LLM and LLM families considered. This paper introduces FaithBench, a summarization hallucination benchmark comprising challenging hallucinations made by 10 modern LLMs from 8 different families, with ground truth annotations by human experts. ``Challenging'' here means summaries on which popular, state-of-the-art hallucination detection models, including GPT-4o-as-a-judge, disagreed on. Our results show GPT-4o and GPT-3.5-Turbo produce the least hallucinations. However, even the best hallucination detection models have near 50\% accuracies on FaithBench, indicating lots of room for future improvement. The repo is https://github.com/vectara/FaithBench

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。