HalluLens: 大语言模型幻觉基准
HalluLens: LLM Hallucination Benchmark
AI总结：
本文提出HalluLens幻觉基准，基于清晰分类法区分外部与内部幻觉，引入动态测试集生成防止数据泄漏，并分析现有基准的局限，旨在推动大语言模型幻觉研究。
摘要：
大语言模型（LLM）经常生成与用户输入或训练数据不一致的响应，这种现象被称为“幻觉”。这些幻觉削弱了用户的信任，并阻碍了生成式人工智能系统的采用。解决幻觉问题对于大语言模型的发展至关重要。本文介绍了一个全面的幻觉基准，该基准基于清晰的幻觉分类法，融合了新的外部幻觉评估任务和现有的内部幻觉评估任务。基准测试幻觉的一个主要挑战是，由于定义和分类不一致，缺乏统一的框架。我们将大语言模型的幻觉与“事实性”区分开来，提出了一种清晰的分类法，区分外部幻觉和内部幻觉，以促进一致性并便于研究。外部幻觉，即生成内容与训练数据不一致，随着大语言模型的发展变得越来越重要。我们的基准包括动态测试集生成，以缓解数据泄漏并确保对此类泄漏的鲁棒性。我们还分析了现有基准，指出了它们的局限性和饱和问题。这项工作旨在：（1）建立清晰的幻觉分类法，（2）引入新的外部幻觉任务，其数据可以动态重新生成以防止因泄漏而饱和，（3）对现有基准进行全面分析，将其与事实性评估区分开来。
英文摘要：
Large language models (LLMs) often generate responses that deviate from user input or training data, a phenomenon known as "hallucination." These hallucinations undermine user trust and hinder the adoption of generative AI systems. Addressing hallucinations is essential for the advancement of LLMs. This paper introduces a comprehensive hallucination benchmark, incorporating both new extrinsic and existing intrinsic evaluation tasks, built upon clear taxonomy of hallucination. A major challenge in benchmarking hallucinations is the lack of a unified framework due to inconsistent definitions and categorizations. We disentangle LLM hallucination from "factuality," proposing a clear taxonomy that distinguishes between extrinsic and intrinsic hallucinations, to promote consistency and facilitate research. Extrinsic hallucinations, where the generated content is not consistent with the training data, are increasingly important as LLMs evolve. Our benchmark includes dynamic test set generation to mitigate data leakage and ensure robustness against such leakage. We also analyze existing benchmarks, highlighting their limitations and saturation. The work aims to: (1) establish a clear taxonomy of hallucinations, (2) introduce new extrinsic hallucination tasks, with data that can be dynamically regenerated to prevent saturation by leakage, (3) provide a comprehensive analysis of existing benchmarks, distinguishing them from factuality evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
