arXiv 2508.087422025-09-24cs.CLcs.AI

SciRerankBench：面向科学检索增强生成大语言模型的排序器基准评测

SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generated LLMs

Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of California, Los Angeles（加州大学洛杉矶分校）；Peking University（北京大学）

Haotian Chen, Qingqing Long, Meng Xiao, Xiao Luo, Wei Ju, Chengrui Wang, Xuezhi Wang, Yuanchun Zhou, Hengshu Zhu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对科学领域RAG-LLMs中排序器评估空白，提出SciRerankBench基准，涵盖五学科、三类Q-C-A对，系统评估13种排序器在五个LLM家族上的性能，揭示其噪声鲁棒性、相关性消歧与事实一致性表现。

摘要：

科学文献问答是迈向新科学发现的关键一步。近年来，两阶段检索增强生成大语言模型（RAG-LLMs）在该领域展现出令人瞩目的进展。这种两阶段框架，尤其是第二阶段（排序器），在科学领域尤为重要，因为术语上的细微差异可能对最终面向事实或知识密集型的答案产生极大的负面影响。尽管取得了显著进展，但这些工作的潜力和局限性仍未得到充分探索。在本工作中，我们提出了一个面向科学排序的RAG基准（SciRerankBench），用于评估RAG-LLMs系统中的排序器，涵盖五个科学学科。为严格评估排序器在噪声鲁棒性、相关性消歧和事实一致性方面的性能，我们构建了三种类型的问题-上下文-答案（Q-C-A）对，即噪声上下文（NC）、语义相似但逻辑不相关的上下文（SSLI）和反事实上下文（CC）。通过对五个LLM家族上的13种广泛使用的排序器进行系统评估，我们提供了关于其相对优势和局限性的详细见解。据我们所知，SciRerankBench是首个专门用于评估RAG-LLMs中排序器的基准，为其未来发展提供了有价值的观察和指导。

英文摘要：

Scientific literature question answering is a pivotal step towards new scientific discoveries. Recently, \textit{two-stage} retrieval-augmented generated large language models (RAG-LLMs) have shown impressive advancements in this domain. Such a two-stage framework, especially the second stage (reranker), is particularly essential in the scientific domain, where subtle differences in terminology may have a greatly negative impact on the final factual-oriented or knowledge-intensive answers. Despite this significant progress, the potential and limitations of these works remain unexplored. In this work, we present a Scientific Rerank-oriented RAG Benchmark (SciRerankBench), for evaluating rerankers within RAG-LLMs systems, spanning five scientific subjects. To rigorously assess the reranker performance in terms of noise resilience, relevance disambiguation, and factual consistency, we develop three types of question-context-answer (Q-C-A) pairs, i.e., Noisy Contexts (NC), Semantically Similar but Logically Irrelevant Contexts (SSLI), and Counterfactual Contexts (CC). Through systematic evaluation of 13 widely used rerankers on five families of LLMs, we provide detailed insights into their relative strengths and limitations. To the best of our knowledge, SciRerankBench is the first benchmark specifically developed to evaluate rerankers within RAG-LLMs, which provides valuable observations and guidance for their future development.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。