arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.04800cs.CLcs.AI
收藏

HoH:用于评估过时信息对检索增强生成影响的动态基准

HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation

State Key Lab of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学)

Jie Ouyang, Tingyue Pan, Mingyue Cheng, Ruiran Yan, Yucong Luo, Jiaying Lin, Qi Liu

AI总结:

本文提出首个专门评估过时信息对RAG影响的动态基准HoH,通过token级差异算法与LLM流水线构建大规模时间知识问答数据集,实验证明过时信息会显著降低RAG准确性并可能误导模型生成有害输出,揭示了RAG面临的时间挑战。

摘要:

尽管检索增强生成(RAG)已成为解决大语言模型(LLM)知识过时问题的有效方法,但它仍面临一个关键挑战:知识库中过时信息的普遍存在。当前研究主要关注纳入最新信息,但检索源中并存的过时信息所产生的影响仍未得到充分解决。为弥补这一空白,我们引入了HoH,这是首个专门设计用于评估过时信息对RAG影响的基准。我们的基准利用token级差异算法与LLM流水线相结合,高效地创建了一个大规模问答数据集,该数据集准确捕捉了现实世界事实中时间知识的演化。通过全面实验,我们揭示了过时信息以两种关键方式显著降低RAG性能:(1)它通过分散模型对正确信息的注意力而大幅降低响应准确性;(2)即使当前信息可用,它也可能误导模型生成潜在有害的输出。当前的RAG方法在处理过时信息时,在检索和生成两个方面都面临困难。这些发现凸显了迫切需要创新解决方案来应对RAG中的时间挑战。我们的代码和数据可在以下地址获取:https://github.com/0russwest0/HoH。

英文摘要:

While Retrieval-Augmented Generation (RAG) has emerged as an effective approach for addressing the knowledge outdating problem in Large Language Models (LLMs), it still faces a critical challenge: the prevalence of outdated information in knowledge bases. Current research primarily focuses on incorporating up-to-date information, yet the impact of outdated information coexisting in retrieval sources remains inadequately addressed. To bridge this gap, we introduce HoH, the first benchmark specifically designed to evaluate the impact of outdated information on RAG. Our benchmark leverages token-level diff algorithms combined with LLM pipelines to efficiently create a large-scale QA dataset that accurately captures the evolution of temporal knowledge in real-world facts. Through comprehensive experiments, we reveal that outdated information significantly degrades RAG performance in two critical ways: (1) it substantially reduces response accuracy by distracting models from correct information, and (2) it can mislead models into generating potentially harmful outputs, even when current information is available. Current RAG approaches struggle with both retrieval and generation aspects when handling outdated information. These findings highlight the urgent need for innovative solutions to address the temporal challenges in RAG. Our code and data are available at: https://github.com/0russwest0/HoH.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑