arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.27752cs.CLcs.SE
收藏

基于分层验证的反向测试用于RAG中的幻觉检测

Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG

Lero, the Research Ireland Centre for Software, University of Limerick(Lero,爱尔兰科学基金会软件研究中心,利默里克大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Ottawa(渥太华大学)

Boxi Yu, Yuzhong Zhang, Liting Lin, Lionel Briand, Emir Muñoz

AI总结:

本文提出RT4CHART框架,通过分层验证提升RAG中上下文忠实度评估的准确性,实现细粒度审计,并在新标注基准上取得最佳检测效果。

摘要:

大型语言模型(LLMs)在检索增强生成(RAG)中持续产生未被支持或与检索上下文冲突的声明。当仅基于检索上下文评估忠实度时,检测此类错误仍具挑战性。现有方法要么提供粗粒度的答案级评分,要么专注于开放领域事实性,往往缺乏细粒度、证据基础的诊断。我们提出RT4CHART,一种用于上下文忠实度评估的反向测试框架。RT4CHART将模型输出分解为可独立验证的声明,并对检索上下文进行分层、局部到全局的验证。每个声明被分配三种标签之一:蕴含、矛盾或无根据。此外,RT4CHART将声明级决策映射回特定答案片段,并从上下文中检索明确的支持或反驳证据,从而实现细粒度和可解释的审计。我们在RAGTruth++(408个样本)和RAGTruth-Enhance(2,675个样本)上评估RT4CHART,一个重新标注的基准。RT4CHART在所有基线中实现了最佳的答案级幻觉检测F1分数。在RAGTruth++上,其F1得分为0.776,优于最强基线83%。在RAGTruth-Enhance上,其片段级F1得分为47.5%。消融研究显示,分层验证设计是性能提升的主要驱动因素。最后,我们的重新标注揭示了比原始标签多1.68倍的幻觉案例,表明现有基准严重低估了幻觉的普遍性。

英文摘要:

Large language models can still hallucinate in retrieval-augmented generation (RAG), producing claims that are unsupported by or conflict with the retrieved context. Detecting such errors remains challenging when faithfulness is judged solely against the retrieved context: many existing detectors return holistic answer-level scores, while others target open-domain factuality or fail to provide evidence-grounded diagnostics. We present RT4CHART, a retromorphic testing framework for context-faithfulness assessment. RT4CHART decomposes an answer into independently verifiable claims, performs hierarchical local-to-global verification against the retrieved context, and assigns each claim one of three labels: entailed, contradicted, or baseless. It further maps these claim-level decisions back to specific answer spans and returns explicit context-side evidence, enabling fine-grained auditing rather than opaque scoring. We evaluate RT4CHART on RAGTruth++ (408 samples) and our re-annotated RAGTruth-Enhance (2,675 samples). RT4CHART achieves the best answer-level hallucination-detection F1 score among the evaluated baselines. On RAGTruth++, it attains a precision of 0.845, a recall of 0.718, and an F1 score of 0.776, representing an 83% relative improvement over the strongest baseline. It also achieves a span-level F1 score of 47.5% on RAGTruth-Enhance. Ablation studies show that claim-based local processing drives most of the observed improvement, while global verification provides selective benefits across datasets. Finally, our re-annotation identifies 1.68X more hallucination cases than the original labels, suggesting that commonly used benchmarks substantially underestimate the prevalence of hallucination.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑