当准确率差距无法认证：审计LLM裁判的跨域重校准

中文摘要 AI 辅助

为LLM裁判在某一任务上拟合的标量重校准映射，在任务分布变化时可能失效，但源-目标准确率差距常被视为该失效的代理指标。我们测试了这一差距在十三个裁判、两个生成器、八个域以及1,176次预声明迁移中能预测什么、能认证什么。在考虑平均分数偏移后，该差距给出了目标校准误差的总体下界，然而相同的差距可能引发相反的迁移结果。精确重要性加权在协变量偏移下恢复了目标期望损失，因此估计加权流程的失败本身并不能确立条件偏移。一个有限样本同时下界证书利用与评估结果不相交的审计标签，将总体界转化为单侧拒绝规则。无泄漏差距相关性为0.25（95%置信区间[-0.09, 0.55]），在第二个生成器上降至0.09，且不支持生成器不变的关联。该证书保持名义覆盖率，但在m=1024时功效仅为0.13，而使用16个标签的目标域温度缩放达到危害率0.09，相比之下源拟合的Platt缩放为0.34。因此，准确率差距是标量概率迁移的弱预警信号，而非部署证书。

英文摘要

A scalar recalibration map fitted for an LLM judge on one task can fail when the task distribution changes, but the source-target accuracy gap is often treated as a proxy for that failure. We test what this gap can predict and what it can certify across thirteen judges, two generators, eight domains, and 1,176 predeclared transfers. After accounting for mean score shift, the gap yields a population lower bound on target calibration error, yet identical gaps can induce opposite transfer outcomes. Exact importance weighting recovers target proper loss under covariate shift, so failure of an estimated weighting pipeline does not by itself establish conditional shift. A finite-sample simultaneous lower certificate converts the population bound into a one-sided rejection rule using audit labels disjoint from evaluation outcomes. The leak-free gap correlation is 0.25 (95% CI [-0.09, 0.55]), falls to 0.09 on the second generator, and does not support a generator-invariant association. The certificate retains nominal coverage but has power 0.13 even at m=1024, whereas target-domain temperature scaling with 16 labels reaches harm rate 0.09, compared with 0.34 for source-fitted Platt scaling. Accuracy gaps are therefore weak warning signals for scalar probability transfer, not deployment certificates.

发表机构

• Kalinga Institute of Industrial Technology（卡林加工业技术学院）
• Iowa State University（爱荷华州立大学）

机构由 AI 辅助整理，请以论文原文为准。