arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.02443cs.CL
收藏

HalluScan:一个用于检测和缓解指令遵循型大语言模型幻觉的系统性基准

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

Sofrecom Tunisia(Sofrecom突尼斯);Orange Innovation(Orange创新)

Ahmed Cherif

AI总结:

提出HalluScan基准框架,通过HalluScore复合指标、自适应检测路由(ADR)和系统错误级联分解,在6种检测方法、4个模型家族和3个领域上评估幻觉检测与缓解,发现NLI验证AUROC最高(0.88),RAV次之(0.66)。

摘要:

大型语言模型(LLM)在各种自然语言处理任务中展现了卓越的能力,但它们仍然容易产生幻觉——生成事实上不正确、与提供上下文不一致或与用户指令不符的内容。我们提出了HalluScan,一个全面的基准框架,系统性地评估了跨越72种配置(涵盖6种检测方法、4个开放权重模型家族和3个不同领域)的幻觉检测与缓解。我们引入了三个关键贡献:(1)HalluScore,一种新颖的复合指标,与人类专家判断的皮尔逊相关系数达到r=0.41;(2)自适应检测路由(ADR),一种智能路由算法,实现了2.0倍的成本降低,仅损失0.1%的AUROC;(3)系统错误级联分解,揭示了不同领域间幻觉错误类型的显著差异。我们的实验表明,NLI验证达到了最高的总体AUROC为0.88,而RAV达到了第二高的AUROC为0.66。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, yet they remain susceptible to hallucinations -- generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive benchmark framework that systematically evaluates hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑