HalluDial：用于自动对话级幻觉评估的大规模基准
HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation
AI总结：
本文提出含4,094段对话、146,856个样本的HalluDial基准，并训练专用裁判模型HalluJudge，以实现对话级事实性与忠实性幻觉的自动评估、定位和依据分析。
摘要：
大语言模型（LLM）显著推动了自然语言处理（NLP）领域的发展，在各类任务上取得卓越表现，并促成了广泛的现实应用。然而，LLM容易产生幻觉，即生成与既有知识相冲突或不忠实于原始来源的内容。现有幻觉基准主要聚焦句子或段落级幻觉检测，忽视了对话级评估、幻觉定位以及依据提供；它们还主要针对事实性幻觉，低估了忠实性幻觉，且往往依赖劳动密集型或非专业化的评估器。为解决这些局限，我们提出 HalluDial，这是首个用于自动对话级幻觉评估的综合性大规模基准。HalluDial 同时涵盖自发生成与诱导产生的幻觉场景，覆盖事实性与忠实性幻觉。该基准包含 4,094 段对话，共计 146,856 个样本。借助 HalluDial，我们对 LLM 在信息检索对话中的幻觉评估能力开展了全面的元评估，并推出专门的裁判语言模型 HalluJudge。HalluDial 的高数据质量使 HalluJudge 在幻觉评估中取得优越或具竞争力的性能，推动了 LLM 对话级幻觉的自动评估，并为理解该现象提供了宝贵洞见。数据集与代码可在 https://github.com/FlagOpen/HalluDial 获取。
英文摘要：
Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP), achieving remarkable performance across diverse tasks and enabling widespread real-world applications. However, LLMs are prone to hallucination, generating content that either conflicts with established knowledge or is unfaithful to the original sources. Existing hallucination benchmarks primarily focus on sentence- or passage-level hallucination detection, neglecting dialogue-level evaluation, hallucination localization, and rationale provision. They also predominantly target factuality hallucinations while underestimating faithfulness hallucinations, often relying on labor-intensive or non-specialized evaluators. To address these limitations, we propose HalluDial, the first comprehensive large-scale benchmark for automatic dialogue-level hallucination evaluation. HalluDial encompasses both spontaneous and induced hallucination scenarios, covering factuality and faithfulness hallucinations. The benchmark includes 4,094 dialogues with a total of 146,856 samples. Leveraging HalluDial, we conduct a comprehensive meta-evaluation of LLMs' hallucination evaluation capabilities in information-seeking dialogues and introduce a specialized judge language model, HalluJudge. The high data quality of HalluDial enables HalluJudge to achieve superior or competitive performance in hallucination evaluation, facilitating the automatic assessment of dialogue-level hallucinations in LLMs and providing valuable insights into this phenomenon. The dataset and the code are available at https://github.com/FlagOpen/HalluDial.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
