关于事实问答中幻觉的早期检测
On Early Detection of Hallucinations in Factual Question Answering
AI总结：
本文研究事实问答中的幻觉早期检测，通过分析LLM的输入、输出和内部状态线索训练二元分类器，实现了高达0.80的AUROC，并发现幻觉前的token已能预测后续幻觉。
摘要：
尽管大型语言模型（LLMs）在帮助人类完成大量任务方面取得了长足进展，但幻觉仍是获取用户信任的主要障碍。即使产生幻觉时，模型生成内容的流畅性和连贯性也使得检测成为一项困难的任务。在这项工作中，我们探讨了与模型生成相关的线索是否能暗示生成内容将包含幻觉。具体而言，我们在开放式问答任务中通过以下方式探测LLMs：1）通过基于Integrated Gradients的token归因分析输入；2）通过Softmax概率分析输出；3）通过自注意力和全连接层激活分析内部状态，以寻找幻觉迹象。我们的结果表明，这些线索的分布在幻觉生成和非幻觉生成之间往往存在差异。基于这一发现，我们训练了二元分类器，将这些线索作为输入特征，把模型生成内容分类为幻觉和非幻觉。这些幻觉分类器实现了高达0.80的AUROC。我们还表明，在幻觉出现之前的token已经能够预测随后发生的幻觉，即使在该幻觉发生之前也是如此。
英文摘要：
While large language models (LLMs) have taken great strides towards helping humans with a plethora of tasks, hallucinations remain a major impediment towards gaining user trust. The fluency and coherence of model generations even when hallucinating makes detection a difficult task. In this work, we explore if the artifacts associated with the model generations can provide hints that the generation will contain hallucinations. Specifically, we probe LLMs at 1) the inputs via Integrated Gradients based token attribution, 2) the outputs via the Softmax probabilities, and 3) the internal state via self-attention and fully-connected layer activations for signs of hallucinations on open-ended question answering tasks. Our results show that the distributions of these artifacts tend to differ between hallucinated and non-hallucinated generations. Building on this insight, we train binary classifiers that use these artifacts as input features to classify model generations into hallucinations and non-hallucinations. These hallucination classifiers achieve up to $0.80$ AUROC. We also show that tokens preceding a hallucination can already predict the subsequent hallucination even before it occurs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
