arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04727cs.IR
收藏

EviRank: 基于证据的LLM排序置信度估计

EviRank: Evidence-Based Confidence Estimation for LLM-Based Ranking

Xidian University(西安电子科技大学)

Meng Yan, Cai Xv, Xujing Wang, Ziyu Guan, Wei Zhao

AI总结:

提出EviRank方法,通过单次前向传播提取三种互补证据并聚合,结合位置感知校准,实现LLM排序的置信度估计与优化。

摘要:

大型语言模型在推荐方面展现出潜力,但由于领域覆盖有限和固有的随机性,引发了可靠性问题。现有的不确定性量化方法存在两个基本挑战:(1)为问答设计的全局置信度分数无法揭示排序列表中哪些位置不可靠;(2)从模型内部提取的细粒度置信度在所有位置上呈现均匀低值,无法过滤不可靠的预测。为解决这些挑战,我们提出了一种基于证据的LLM排序置信度估计方法(EviRank)。我们从单次前向传播中提取三种互补证据,并通过可靠的证据聚合进行整合。此外,我们认识到排序位置本质上是不平等的,并引入位置感知校准。最后,校准后的置信度指导排序优化。在三个数据集上的实验表明,我们的方法在推荐和不确定性量化方面均达到了最先进的性能。

英文摘要:

Large Language Models show promise for recommendation, but they raise reliability concerns due to limited domain coverage and inherent stochasticity. Existing uncertainty quantification methods persist two fundamental challenges: (1) the global confidence score designed for question answering fails to reveal which positions are unreliable in ranking list; (2) fine-grained confidence extracted from model internals exhibits uniformly low values across all positions, making it impossible to filter unreliable predictions. To tackle the challenges, we propose an evidence-based confidence estimation for LLM-based ranking (EviRank). We extract three complementary evidences from a single forward pass and aggregate them via reliable opinion aggregation. Furthermore, we recognize that ranking positions are inherently unequal, and introduce a position-aware calibration. Lastly, the calibrated confidence guides ranking optimization. Experiments on three datasets demonstrate that our method achieves state-of-the-art performance on both recommendation and uncertainty quantification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑