arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.01409cs.CL
收藏

LiveSearchBench:一个面向动态知识检索与推理的自动构建基准

LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge

University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);London School of Economics(伦敦经济学院);BUPT(北京邮电大学);Harbin Institute of Technology(哈尔滨工业大学);SUTD(新加坡科技设计大学);University of Oxford(牛津大学)

Heng Zhou, Ao Yu, Yuchen Fan, Jianing Shi, Li Kang, Hejia Geng, Yongting Zhang, Yutao Fan, Yuhao Wu, Tiancheng He, Yiran Qin, Lei Bai, Zhenfei Yin

AI总结:

LiveSearchBench 通过自动对比 Wikidata 快照差异并生成多难度可验证问题,构建动态知识检索与推理基准,揭示 LLMs 在预训练后新事实上的性能下降及检索增强方法的不足。

摘要:

在问答任务上评估大语言模型(LLMs)通常依赖静态基准,这类基准奖励记忆能力而低估检索的作用,无法反映世界知识的动态特性。我们提出 LiveSearchBench,一个从近期知识更新中自动构建检索依赖型基准的流水线。我们的方法计算连续 Wikidata 快照之间的差异,对候选三元组进行质量过滤,并在三个推理难度层级上合成自然语言问题,每个问题都通过 SPARQL 验证保证具有唯一且可验证的答案。该流水线完全自动化,可跨时间扩展,并最大限度地减少人工干预,从而能够持续再生成具有时间锚定的基准。实验表明,当模型面对晚于预训练时间的事实知识时,性能出现明显下降,且这一差距在多跳查询上最为显著。检索增强方法和更大规模的指令微调模型带来了部分提升,但未能弥合这一时效性差距。通过设计,LiveSearchBench 将评估重心从静态记忆转向需要最新检索与推理的任务,为在知识不断演化的条件下对 LLMs 进行系统性、长期性评估奠定了基础。

英文摘要:

Evaluating large language models (LLMs) on question answering often relies on static benchmarks that reward memorization and understate the role of retrieval, failing to capture the dynamic nature of world knowledge. We present LiveSearchBench, an automated pipeline for constructing retrieval-dependent benchmarks from recent knowledge updates. Our method computes deltas between successive Wikidata snapshots, filters candidate triples for quality, and synthesizes natural-language questions at three levels of reasoning difficulty, each guaranteed to admit a unique, verifiable answer through SPARQL validation. The pipeline is fully automated, scalable across time, and minimizes human intervention, enabling continual regeneration of temporally grounded benchmarks. Experiments show a pronounced performance drop when models confront facts that post-date pretraining, with the gap most salient on multi-hop queries. Retrieval augmented methods and larger, instruction-tuned models provide partial gains but fail to close this recency gap. By design, LiveSearchBench shifts evaluation from static memorization toward tasks that require up-to-date retrieval and reasoning, offering a foundation for systematic, long-term assessment of LLMs under evolving knowledge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑