基于演进排行榜的RAG中LLM忠实度基准测试
Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards
AI总结：
本文提出FaithJudge框架，利用人工标注幻觉示例改进RAG中LLM幻觉的自动评估，并构建增强型排行榜，在摘要、问答和数据到文本任务上更可靠地基准测试LLM忠实度。
摘要：
检索增强生成（RAG）旨在通过将回答建立在外部上下文之上来减少幻觉，然而即使提供了相关上下文，大型语言模型（LLMs）仍经常引入无依据的信息或产生矛盾。本文介绍了Vectara在测量和基准测试RAG中LLM忠实度方面的两项互补工作。首先，我们描述了我们最初的幻觉排行榜，该排行榜自2023年以来一直使用我们的HHEM幻觉检测模型跟踪LLM的幻觉率。受当前幻觉检测方法中观察到的局限性的启发，我们引入了FaithJudge，这是一个LLM-as-a-judge框架，利用多样的人工标注幻觉示例池来大幅改进LLM的自动化幻觉评估。我们推出了一个以FaithJudge为核心的增强型幻觉排行榜，在摘要生成、问答和数据到文本生成任务上对LLM的RAG忠实度进行基准测试。FaithJudge使得对RAG中LLM幻觉的基准测试更加可靠，并支持开发更值得信赖的生成式AI系统：https://github.com/vectara/FaithJudge。
英文摘要：
Retrieval-augmented generation (RAG) aims to reduce hallucinations by grounding responses in external context, yet large language models (LLMs) still frequently introduce unsupported information or contradictions even when provided with relevant context. This paper presents two complementary efforts at Vectara to measure and benchmark LLM faithfulness in RAG. First, we describe our original hallucination leaderboard, which has tracked hallucination rates for LLMs since 2023 using our HHEM hallucination detection model. Motivated by limitations observed in current hallucination detection methods, we introduce FaithJudge, an LLM-as-a-judge framework that leverages a pool of diverse human-annotated hallucination examples to substantially improve the automated hallucination evaluation of LLMs. We introduce an enhanced hallucination leaderboard centered on FaithJudge that benchmarks LLMs on RAG faithfulness in summarization, question-answering, and data-to-text generation tasks. FaithJudge enables a more reliable benchmarking of LLM hallucinations in RAG and supports the development of more trustworthy generative AI systems: https://github.com/vectara/FaithJudge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
