arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.05073stat.ME
收藏

异质性裁判感知排序与敏感性、分歧和置信度

Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence

The University of Hong Kong(香港大学)

Shibo Yu, Yingzhou Wang, Yan Chen, Guodong Li, Jin-Hong Du

AI总结:

本文提出HJA排序框架,通过分离共识排名、裁判敏感性和残差偏好分歧,提升多裁判比较数据的恢复、鲁棒性和不确定性校准能力。

摘要:

多裁判的成对比较是大语言模型评估和偏好建模的核心,但标准排序流程常将判断合并为单一评分向量,将系统性裁判分歧视为噪声。我们提出异质性裁判感知(HJA)排序,一种结构化的多裁判排序框架,将共识排名、裁判特定的敏感性和残余偏好分歧分离。HJA将排序、裁判敏感性和结构化分歧视为独立的推断目标。我们建立了这种分解可识别的条件,并开发了一种锚定交替算法,以保持可识别的几何结构。对于置信度量化,我们研究了一种固定面板重复比较制度,在此制度中,裁判面板可能保持固定或适度,而信息通过重复判断增长。这产生了对共识和裁判特定排名对比、敏感性参数、成对概率以及残余分歧总结的不确定性声明。在合成和真实多裁判比较数据上的实验表明,HJA相比合并和敏感性-only基线在恢复、鲁棒性、不确定性校准和近平局性能上有所提升。拟合模型还提供裁判分歧和模型亲和模式的诊断,给出在异质比较判断下的统计学基础框架。

英文摘要:

Pairwise comparisons from multiple judges are central to large language model evaluation and preference modeling, yet standard ranking pipelines often pool judgments into a single score vector, treating systematic judge disagreement as noise. We propose Heterogeneous Judge-Aware (HJA) ranking, a structured multi-judge ranking framework that separates consensus ranking, judge-specific sensitivity to consensus, and residual preference disagreement. HJA thereby treats ranking, judge sensitivity, and structured disagreement as separate inferential targets. We establish conditions under which this decomposition is identifiable and develop an anchored alternating algorithm that preserves the identifying geometry. For confidence quantification, we study a fixed-panel repeated-comparison regime in which the judge panel may remain fixed or modest while information grows through repeated judgments. This yields uncertainty statements for consensus and judge-specific ranking contrasts, sensitivity parameters, pairwise probabilities, and summaries of residual disagreement. Experiments on synthetic and real multi-judge comparison data show that HJA improves recovery, robustness, uncertainty calibration, and near-tie performance relative to pooled and sensitivity-only baselines. The fitted model also provides diagnostics for judge disagreement and model-affinity patterns, giving a statistically grounded framework for ranking under heterogeneous comparative judgments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑