AutoHall: 自动化生成大语言模型的事实性幻觉数据集
AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models
AI总结：
AutoHall通过自动化生成模型特定的幻觉数据集，提升大语言模型的事实性内容检测能力。
摘要：
大语言模型（LLMs）在各种领域中获得了广泛应用，但仍然在生成事实性内容时面临幻觉问题。目前，事实性内容中幻觉频繁发生，对可信的LLMs构成巨大挑战。然而，幻觉检测受到手动标注幻觉内容的劳动密集和昂贵的限制。同时，由于不同的LLMs表现出不同的幻觉类型和发生率，幻觉数据集的收集本质上是模型特定的，这也增加了成本。为了解决这个问题，本文提出了一种称为AutoHall的方法，该方法基于现有的事实核查数据集，自动构建模型特定的幻觉数据集。实证结果揭示了不同模型之间幻觉比例和类型的差异。此外，我们引入了一种零资源和黑盒的幻觉检测方法，基于自矛盾性来识别我们构建的数据集中的幻觉，其检测性能优于基线方法。对我们的数据集的进一步分析提供了关于LLM幻觉可能因素的见解。我们的代码和数据集在https://github.com/zouyingcao/AutoHall上公开可用。
英文摘要：
Large language models (LLMs) have gained broad applications across various domains but still struggle with hallucinations. Currently, hallucinations occur frequently in the generation of factual content and pose a great challenge to trustworthy LLMs. However, hallucination detection is hindered by the laborious and expensive manual annotation of hallucinatory content. Meanwhile, as different LLMs exhibit distinct types and rates of hallucination, the collection of hallucination datasets is inherently model-specific, which also increases the cost. To address this issue, this paper proposes a method called $\textbf{AutoHall}$ for $\underline{Auto}$matically constructing model-specific $\underline{Hall}$ucination datasets based on existing fact-checking datasets. The empirical results reveal variations in hallucination proportions and types among different models. Moreover, we introduce a zero-resource and black-box hallucination detection method based on self-contradiction to recognize the hallucination in our constructed dataset, achieving superior detection performance compared to baselines. Further analysis on our dataset provides insight into factors that may contribute to LLM hallucinations. Our codes and datasets are publicly available at https://github.com/zouyingcao/AutoHall.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
