AuthorityBench: 评估LLM权威感知以实现可靠的检索增强生成
AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation
AI总结:
本文提出AuthorityBench基准,通过三个数据集评估LLM权威感知能力,发现ListJudge和PairJudge方法与真实权威最相关,且权威感知对检索增强生成的准确性有显著提升。
摘要:
检索增强生成(RAG)通过外部知识增强大型语言模型(LLMs),但易受低权威来源影响。本文研究LLM是否能感知信息权威,提出AuthorityBench基准,包含DomainAuth、EntityAuth和RAGAuth三个数据集。通过五种LLM和三种判断方法评估,结果显示ListJudge和PairJudge方法与真实权威最相关,且ListJudge成本效益最佳。下游实验表明,权威引导过滤显著提升回答准确性,验证了权威感知在可靠知识检索中的重要性。代码和基准见:https://github.com/Trustworthy-Information-Access/AuthorityBench.
英文摘要:
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) with external knowledge but remains vulnerable to low-authority sources that can propagate misinformation. We investigate whether LLMs can perceive information authority - a capability extending beyond semantic understanding. To address this, we introduce AuthorityBench, a comprehensive benchmark for evaluating LLM authority perception comprising three datasets: DomainAuth (10K web domains with PageRank-based authority), EntityAuth (22K entities with popularity-based authority), and RAGAuth (120 queries with documents of varying authority for downstream evaluation). We evaluate five LLMs using three judging methods (PointJudge, PairJudge, ListJudge) across multiple output formats. Results show that ListJudge and PairJudge with PointScore output achieve the strongest correlation with ground-truth authority, while ListJudge offers optimal cost-effectiveness. Notably, incorporating webpage text consistently degrades judgment performance, suggesting authority is distinct from textual style. Downstream experiments on RAG demonstrate that authority-guided filtering largely improves answer accuracy, validating the practical importance of authority perception for reliable knowledge retrieval. Code and benchmark are available at: https://github.com/Trustworthy-Information-Access/AuthorityBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。