LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems
organization= School of Computer Science;Technology, Dalian University of Technology , addressline= No.2 Linggong Road, Ganjingzi District , city= Dalian , postcode= 116024 , country= China;organization= School of Information Engineering, Liaodong University , addressline= No.116 Linjiang Back Street, Zhenan District , city= Dandong , postcode= 118001 , country= China;organization= School of Information Engineering, Dalian Ocean University , addressline= No. 2-52, Heishijiao Street, Shahekou District , city= Dalian , postcode= 116023 , country= China;organization= Information Technology Center, Qinghai University , addressline= 251 Ningda Road, Chengbei District , city= Xining , postcode= 810016 , country= China;organization= School of Electronic;Information Engineering, Liaoning Technical University , addressline= 188 Longwan South Street, Sijiatun District , city= Huludao , postcode= 125105 , country= China;organization= School of Artificial Intelligence, Tianjin Normal University , addressline= 393 Binshui West Road, Xiqing District , city= Tianjin , postcode= 300387 , country= China;organization= Tencent (Dalian Northern Interactive Entertainment Technology Co., Ltd.) , addressline= 21/F, Tencent Building, No. 26 Jingxian St, Ganjingzi District , city= Dalian , postcode= 116085 , country= China
Large language models (LLMs) have revolutionized natural language processing, yet hallucinations in knowledge-intensive tasks remain a critical challenge. Retrieval-augmented generation (RAG) addresses this by integrating external knowledge, but its efficacy depends on accurate document retrieval and ranking. Although existing rerankers demonstrate effectiveness, they frequently necessitate specialized training, impose substantial computational expenses, and fail to fully exploit the semantic capabilities of LLMs, particularly their inherent confidence signals. We propose the LLM-Confidence Reranker (LCR), a training-free, plug-and-play algorithm that enhances reranking in RAG systems by leveraging black-box LLM confidence derived from Maximum Semantic Cluster Proportion (MSCP). LCR employs a two-stage process: confidence assessment via multinomial sampling and clustering, followed by binning and multi-level sorting based on query and document confidence thresholds. This approach prioritizes relevant documents while preserving original rankings for high-confidence queries, ensuring robustness. Evaluated on BEIR and TREC benchmarks with BM25 and Contriever retrievers, LCR--using only 7--9B-parameter pre-trained LLMs--consistently improves NDCG@5 by up to 20.6% across pre-trained LLM and fine-tuned Transformer rerankers, without degradation. Ablation studies validate the hypothesis that LLM confidence positively correlates with document relevance, elucidating LCR's mechanism. LCR offers computational efficiency, parallelism for scalability, and broad compatibility, mitigating hallucinations in applications like medical diagnosis.
SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of California, Los Angeles(加州大学洛杉矶分校);Peking University(北京大学)
Verbal-R3: Verbal Reranker as the Missing Bridge between Retrieval and Reasoning
Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电子与计算机工程系);Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔国立大学人工智能跨学科项目);AIIS, ASRI, INMC, and ISRC, Seoul National University(首尔国立大学人工智能研究所、人工智能研究室、智能网络研究中心和智能系统研究中心);DGIST(延世大学)
Less is More: Compact Clue Selection for Efficient Retrieval-Augmented Generation Reasoning
School of Artificial Intelligence, Beihang University(北航人工智能学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Computer Science and Engineering, Beihang University(北航计算机科学与工程学院);Beijing Academy of Blockchain and Edge Computing(北京区块链与边缘计算研究院)
Optimizing Retrieval-Augmented Generation of Medical Content for Spaced Repetition Learning
Poznań University of Medical Sciences(波兹南医科大学);Adam Mickiewicz University(亚当·密茨凯维奇大学);Wydawnictwo Naukowe PWN(PWN科学出版社);SuperMemo World(SuperMemo世界公司);Association for Research and Applications of Artificial Intelligence(人工智能研究与应用协会)
CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation
School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Information Engineering, Dalian Ocean University(大连海洋大学信息工程学院);School of Information Engineering, Liaodong University(辽东学院信息工程学院);Information Technology Center, Qinghai University(青海大学信息技术中心);College of Health-Preservation and Wellness, Dalian Medical University(大连医科大学养生与健康学院);Tencent(腾讯)
Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment
Elmore Family School of Electrical and Computer Engineering, Purdue University(普渡大学埃尔莫尔电气与计算机工程学院);Yonsei University(延世大学);University at Buffalo–SUNY(纽约州立大学布法罗分校)
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
Can LLM Rerankers Predict Their Own Ranking Performance?
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);City University of Hong Kong(香港城市大学);Tencent Technology(腾讯科技);Peking University(北京大学)