arXiv 2603.192812026-03-02cs.CLcs.AIcs.IR

URAG：一种用于检索增强大型语言模型不确定性量化的基准

URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models

Uppsala University（乌普萨拉大学）；University of Science, VNU-HCM（VNU-HCM大学）；Indiana University（印第安纳大学）；FPT Software, AI Center（FPT软件人工智能中心）；The Pennsylvania State University（宾夕法尼亚州立大学）；VNU University of Engineering（VNU工程大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）

Vinh Nguyen, Cuong Dang, Jiahao Zhang, Hoa Tran, Minh Tran, Trinh Chau, Thai Le, Lu Cheng, Suhang Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

URAG通过构建多选问答任务评估RAG系统在医疗、编程、科学、数学等领域的不确定性，揭示准确性与不确定性的关系及不同RAG方法的性能差异。

摘要：

URAG通过构建多选问答任务评估RAG系统在医疗、编程、科学、数学等领域的不确定性，揭示准确性与不确定性的关系及不同RAG方法的性能差异。

英文摘要：

Retrieval-Augmented Generation (RAG) has emerged as a widely adopted approach for enhancing LLMs in scenarios that demand extensive factual knowledge. However, current RAG evaluations concentrate primarily on correctness, which may not fully capture the impact of retrieval on LLM uncertainty and reliability. To bridge this gap, we introduce URAG, a comprehensive benchmark designed to assess the uncertainty of RAG systems across various fields like healthcare, programming, science, math, and general text. By reformulating open-ended generation tasks into multiple-choice question answering, URAG allows for principled uncertainty quantification via conformal prediction. We apply the evaluation pipeline to 8 standard RAG methods, measuring their performance through both accuracy and prediction-set sizes based on LAC and APS metrics. Our analysis shows that (1) accuracy gains often coincide with reduced uncertainty, but this relationship breaks under retrieval noise; (2) simple modular RAG methods tend to offer better accuracy-uncertainty trade-offs than more complex reasoning pipelines; and (3) no single RAG approach is universally reliable across domains. We further show that (4) retrieval depth, parametric knowledge dependence, and exposure to confidence cues can amplify confident errors and hallucinations. Ultimately, URAG establishes a systematic benchmark for analyzing and enhancing the trustworthiness of retrieval-augmented systems. Our code is available on GitHub.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。