EviRank: 基于证据的LLM排序置信度估计
EviRank: Evidence-Based Confidence Estimation for LLM-Based Ranking
AI总结:
提出EviRank方法,通过单次前向传播提取三种互补证据并聚合,结合位置感知校准,实现LLM排序的置信度估计与优化。
摘要:
大型语言模型在推荐方面展现出潜力,但由于领域覆盖有限和固有的随机性,引发了可靠性问题。现有的不确定性量化方法存在两个基本挑战:(1)为问答设计的全局置信度分数无法揭示排序列表中哪些位置不可靠;(2)从模型内部提取的细粒度置信度在所有位置上呈现均匀低值,无法过滤不可靠的预测。为解决这些挑战,我们提出了一种基于证据的LLM排序置信度估计方法(EviRank)。我们从单次前向传播中提取三种互补证据,并通过可靠的证据聚合进行整合。此外,我们认识到排序位置本质上是不平等的,并引入位置感知校准。最后,校准后的置信度指导排序优化。在三个数据集上的实验表明,我们的方法在推荐和不确定性量化方面均达到了最先进的性能。
英文摘要:
Large Language Models show promise for recommendation, but they raise reliability concerns due to limited domain coverage and inherent stochasticity. Existing uncertainty quantification methods persist two fundamental challenges: (1) the global confidence score designed for question answering fails to reveal which positions are unreliable in ranking list; (2) fine-grained confidence extracted from model internals exhibits uniformly low values across all positions, making it impossible to filter unreliable predictions. To tackle the challenges, we propose an evidence-based confidence estimation for LLM-based ranking (EviRank). We extract three complementary evidences from a single forward pass and aggregate them via reliable opinion aggregation. Furthermore, we recognize that ranking positions are inherently unequal, and introduce a position-aware calibration. Lastly, the calibrated confidence guides ranking optimization. Experiments on three datasets demonstrate that our method achieves state-of-the-art performance on both recommendation and uncertainty quantification.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。