谁漂移了：系统还是裁判？LLM评估流水线中的随时有效归因
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
AI总结：
提出一种基于固定锚点集和赌检验的方法，区分LLM评估中产品性能下降与裁判模型变化导致的分数漂移，并证明其随时有效性和归因准确性。
摘要：
对LLM产品的持续评估依赖于一个被视为地面真相的强大LLM裁判：一个廉价的监控器对每次交互进行评分，当分数下降时团队会收到警报。但裁判本身是一个API背后的模型，静默的版本升级或评分提示更新会改变其评分方式——因此每次漂移警报在更差的产品和变化的裁判之间是模糊的。我们通过一个固定的人工标注锚点集（当前裁判以稳定间隔重新评分）、一个关于裁判与人类差距的二次赌e过程，以及一个返回{无, 系统, 裁判}判决的守卫窗口规则来解决这种模糊性。我们证明了随时有效性、单向识别（只有裁判可以移动锚点）、一个归因竞赛（其设计法则是锚点必须跑赢它们守卫的主过程）以及过程正交性。在两个真实的裁判变化中，静默版本升级在60/60次运行中被检测为裁判漂移，且零次误归因为系统；而一个污染性的严格提示变化在守卫宽度为300时，120次运行中有110次被正确归因——而行业默认的滚动z检验在75%的无漂移流上产生误报。每个实验在第二个领域（TL;DR摘要）上重复，无需重新调整参数，并且当领域不同时，差异正是竞赛所预测的：严格提示变化在那里更强烈地改变分数，因此锚点触发更快，归因变得完美（240/240）。该监控器的运行成本约为对每个项目使用强裁判的0.64倍，或在更便宜但更聋的模式下为0.21倍。
英文摘要：
Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
