arXiv 2602.131102026-02cs.CLcs.AI

SCOPE: 选择性保形优化的成对LLM评判

SCOPE: Selective Conformal Optimized Pairwise LLM Judging

Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada（达勒豪西大学计算机科学学院）；Department of Computer Science, Emory University, Atlanta, GA, USA（埃默里大学计算机科学系）

Sher Badshah, Ali Emami, Hassan Sajjad

备注：: Accepted at ICML 2026. 23 pages (9 main plus appendix), 7 figures, 11 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出SCOPE框架，通过校准接受阈值控制非弃权判断的错误率，并引入双向偏好熵（BPE）提供无偏不确定性信号，实现可靠且高覆盖率的LLM成对评估。

摘要：

大型语言模型（LLM）越来越多地被用作成对评估中的可扩展评判者，但它们仍然容易受到校准偏差和偏见的影响。我们提出SCOPE（选择性保形优化成对评估），一个校准接受阈值的框架，使得在可交换性条件下，非弃权判断的错误率最多为用户指定的水平$α$。为了向SCOPE提供无偏的不确定性信号，我们引入了双向偏好熵（BPE），它在两个响应位置下查询评判者，并将顺序平均的偏好概率转换为基于熵的分数。在各种成对评判基准上，BPE在校准和区分能力方面优于标准置信度代理，而SCOPE始终满足目标风险界限（在$α=0.10$时，经验FDR约为0.097至0.099），并保持较高的覆盖率。与原始基线相比，在相同风险约束下，SCOPE接受的判断数量最多增加2.4倍，表明BPE能够实现可靠且高覆盖率的基于LLM的评估。

英文摘要：

Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose \textsc{Scope} (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level $α$. To supply \textsc{Scope} with a bias-neutral uncertainty signal, we introduce Bidirectional Preference Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an entropy-based score. Across various pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while \textsc{Scope} consistently satisfies the target risk bound (empirical FDR $\approx 0.097$--$0.099$ at $α=0.10$) and retains substantial coverage. Compared to vanilla baselines, \textsc{Scope} accepts up to $2.4\times$ more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文