检测和纠正商业大语言模型和深度研究代理中的参考幻觉
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
AI总结：
研究通过评估10种模型和代理在DRBench和ExpertQA上的引用URL可靠性，发现3-13%的URL存在幻觉，5-18%无法解析。提出urlhealth工具可有效减少非解析URL，验证引用URL的有效性与可纠正性。
摘要：
大型语言模型和深度研究代理提供引用URL以支持其主张，但这些引用的可靠性尚未系统测量。我们通过DRBench（53,090个URL）和ExpertQA（168,021个跨32个学术领域的URL）使用10种模型和3种模型，探讨了六个关于引用URL有效性的研究问题。发现3-13%的引用URL是幻觉——在Wayback Machine中无记录且可能从未存在，而5-18%是无法解析的。深度研究代理生成的引用数量比搜索增强型LLM多，但幻觉URL的比例更高。领域效应显著：非解析率从Business的5.4%到Theology的11.4%，且每种模型的效果更大。分解失败发现，某些模型伪造所有非解析URL，而其他模型显示大量链接过期比例，表明真实检索。作为解决方案，我们发布urlhealth，一个开源工具，利用Wayback Machine检查URL活力并分类过时与幻觉URL。在代理自我纠正实验中，配备urlhealth的模型将非解析引用URL减少6-79倍至低于1%，但效果取决于模型的工具使用能力。该工具和所有数据均公开可用。我们的特征分析、失败分类和开源工具表明，引用URL的有效性在大规模上可测量且在实践中可纠正。
英文摘要：
Large language models and deep research agents supply citation URLs to support their claims, yet the reliability of these citations has not been systematically measured. We address six research questions about citation URL validity using 10 models and agents on DRBench (53,090 URLs) and 3 models on ExpertQA (168,021 URLs across 32 academic fields). We find that 3--13\% of citation URLs are hallucinated -- they have no record in the Wayback Machine and likely never existed -- while 5--18\% are non-resolving overall. Deep research agents generate substantially more citations per query than search-augmented LLMs but hallucinate URLs at higher rates. Domain effects are pronounced: non-resolving rates range from 5.4\% (Business) to 11.4\% (Theology), with per-model effects even larger. Decomposing failures reveals that some models fabricate every non-resolving URL, while others show substantial link-rot fractions indicating genuine retrieval. As a solution, we release urlhealth, an open-source tool for URL liveness checking and stale-vs-hallucinated classification using the Wayback Machine. In agentic self-correction experiments, models equipped with urlhealth reduce non-resolving citation URLs by $6\textrm{--}79\times$ to under 1\%, though effectiveness depends on the model's tool-use competence. The tool and all data are publicly available. Our characterization findings, failure taxonomy, and open-source tooling establish that citation URL validity is both measurable at scale and correctable in practice.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
