DiaHalu：面向大语言模型的对话级幻觉评估基准
DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Models
AI总结：
针对现有幻觉基准多为诱导生成、仅关注事实性幻觉且缺乏对话级设置的问题，该研究提出首个对话级幻觉评估基准DiaHalu，覆盖多领域多幻觉子类型，实验验证其具有挑战性与研究价值。
摘要：
近年来大语言模型（LLM）虽取得显著进展，但幻觉问题仍是一大挑战，目前已有众多基准被提出用于检测幻觉。然而，部分基准中的幻觉并非LLM自然生成，而是被刻意诱导产生的；同时许多基准仅关注事实性幻觉，却忽略了忠实性幻觉。此外，尽管对话模式在LLM时代应用更为广泛，但现有基准仅聚焦于句子级和段落级幻觉。本研究中，我们提出了据我们所知首个对话级幻觉评估基准DiaHalu。首先，我们将收集到的主题整合进系统提示词，让两个ChatGPT3.5进行对话；随后，我们人工修改不符合人类语言习惯的内容，再交由LLM重新生成，模拟真实的人机交互场景；最后由专业学者对数据集中的所有样本进行标注。DiaHalu涵盖四个常见的多轮对话领域，以及从事实性和忠实性幻觉延伸出的五种幻觉子类型。在该数据集上对多款知名LLM及检测方法开展的实验表明，DiaHalu是一个具有挑战性的基准，对后续研究具有重要价值。
英文摘要：
Since large language models (LLMs) achieve significant success in recent years, the hallucination issue remains a challenge, numerous benchmarks are proposed to detect the hallucination. Nevertheless, some of these benchmarks are not naturally generated by LLMs but are intentionally induced. Also, many merely focus on the factuality hallucination while ignoring the faithfulness hallucination. Additionally, although dialogue pattern is more widely utilized in the era of LLMs, current benchmarks only concentrate on sentence-level and passage-level hallucination. In this study, we propose DiaHalu, the first dialogue-level hallucination evaluation benchmark to our knowledge. Initially, we integrate the collected topics into system prompts and facilitate a dialogue between two ChatGPT3.5. Subsequently, we manually modify the contents that do not adhere to human language conventions and then have LLMs re-generate, simulating authentic human-machine interaction scenarios. Finally, professional scholars annotate all the samples in the dataset. DiaHalu covers four common multi-turn dialogue domains and five hallucination subtypes, extended from factuality and faithfulness hallucination. Experiments through some well-known LLMs and detection methods on the dataset show that DiaHalu is a challenging benchmark, holding significant value for further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
