arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13571cs.CLcs.AI
收藏

LLM-Confidence Reranker: 一种无需训练的增强检索增强生成系统的方法

LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems

organization= School of Computer Science;Technology, Dalian University of Technology , addressline= No.2 Linggong Road, Ganjingzi District , city= Dalian , postcode= 116024 , country= China;organization= School of Information Engineering, Liaodong University , addressline= No.116 Linjiang Back Street, Zhenan District , city= Dandong , postcode= 118001 , country= China;organization= School of Information Engineering, Dalian Ocean University , addressline= No. 2-52, Heishijiao Street, Shahekou District , city= Dalian , postcode= 116023 , country= China;organization= Information Technology Center, Qinghai University , addressline= 251 Ningda Road, Chengbei District , city= Xining , postcode= 810016 , country= China;organization= School of Electronic;Information Engineering, Liaoning Technical University , addressline= 188 Longwan South Street, Sijiatun District , city= Huludao , postcode= 125105 , country= China;organization= School of Artificial Intelligence, Tianjin Normal University , addressline= 393 Binshui West Road, Xiqing District , city= Tianjin , postcode= 300387 , country= China;organization= Tencent (Dalian Northern Interactive Entertainment Technology Co., Ltd.) , addressline= 21/F, Tencent Building, No. 26 Jingxian St, Ganjingzi District , city= Dalian , postcode= 116085 , country= China

Zhipeng Song, Xiangyu Kong, Xinrui Bao, Yizhi Zhou, Jiulong Jiao, Sitong Liu, Yuhang Zhou, Heng Qi

AI总结:

LLM-Confidence Reranker通过利用LLM置信度信号,无需训练提升检索增强生成系统的排序效果,有效减少幻觉问题。

摘要:

大型语言模型(LLMs)已彻底改变自然语言处理,但在知识密集型任务中仍存在幻觉问题。检索增强生成(RAG)通过整合外部知识来解决这一问题,但其效果依赖于准确的文档检索和排序。尽管现有排序器显示出有效性,但它们通常需要专门的训练,计算成本高,并未能充分利用LLM的语义能力,特别是其内在的置信度信号。我们提出LLM-Confidence Reranker(LCR),一种无需训练、即插即用的算法,通过利用最大语义聚类比例(MSCP)从黑盒LLM中提取置信度来增强RAG系统的排序。LCR采用两阶段过程:通过多项分布采样和聚类进行置信度评估,随后基于查询和文档置信度阈值进行分箱和多级排序。该方法优先选择相关文档,同时在高置信度查询中保留原始排名,确保鲁棒性。在BEIR和TREC基准测试中,使用仅7-9B参数预训练LLM的LCR,在预训练LLM和微调Transformer排序器上,NDCG@5得分提高最高20.6%,且无下降。消融研究验证了LLM置信度与文档相关性正相关的假设,阐明了LCR的机制。LCR提供计算效率、并行扩展性和广泛兼容性,有助于减少医疗诊断等应用中的幻觉问题。

英文摘要:

Large language models (LLMs) have revolutionized natural language processing, yet hallucinations in knowledge-intensive tasks remain a critical challenge. Retrieval-augmented generation (RAG) addresses this by integrating external knowledge, but its efficacy depends on accurate document retrieval and ranking. Although existing rerankers demonstrate effectiveness, they frequently necessitate specialized training, impose substantial computational expenses, and fail to fully exploit the semantic capabilities of LLMs, particularly their inherent confidence signals. We propose the LLM-Confidence Reranker (LCR), a training-free, plug-and-play algorithm that enhances reranking in RAG systems by leveraging black-box LLM confidence derived from Maximum Semantic Cluster Proportion (MSCP). LCR employs a two-stage process: confidence assessment via multinomial sampling and clustering, followed by binning and multi-level sorting based on query and document confidence thresholds. This approach prioritizes relevant documents while preserving original rankings for high-confidence queries, ensuring robustness. Evaluated on BEIR and TREC benchmarks with BM25 and Contriever retrievers, LCR--using only 7--9B-parameter pre-trained LLMs--consistently improves NDCG@5 by up to 20.6% across pre-trained LLM and fine-tuned Transformer rerankers, without degradation. Ablation studies validate the hypothesis that LLM confidence positively correlates with document relevance, elucidating LCR's mechanism. LCR offers computational efficiency, parallelism for scalability, and broad compatibility, mitigating hallucinations in applications like medical diagnosis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑