成为经验丰富的法官：用于评估者的选择性测试时间学习
Becoming Experienced Judges: Selective Test-Time Learning for Evaluators
AI总结：
本文提出选择性测试时间学习框架，使评估者在推理过程中逐步改进，通过自适应元提示提升评估效果，实验证明其在多个基准上的优越性。
摘要：
在推理和对齐任务中，大型语言模型（LLM）作为法官已成为标准。尽管在部署中评估了许多样本，这些评估者通常（i）独立处理每个案例，错失积累经验的机会，（ii）依赖于单一固定提示处理所有案例，忽视了需要针对样本特定的评估标准。我们介绍了在评估中学习（LWE）框架，该框架允许评估者在推理时间逐步改进，而无需训练或验证集。LWE维护一个不断发展元提示，（i）生成针对样本特定的评估指示，（ii）通过自动生成的反馈进行自我优化。此外，我们提出了选择性LWE，它仅在自不一致的情况下更新元提示，将计算集中在最关键的地方。这种选择性方法保留了顺序学习的好处，同时成本显著降低。在两个成对比较基准上，选择性LWE优于强大的基线，实证表明评估者可以在顺序测试中通过简单的选择性更新改进，从他们挣扎的案例中学习最多。
英文摘要：
Automatic evaluation with large language models, commonly known as LLM-as-a-judge, is now standard across reasoning and alignment tasks. Despite evaluating many samples in deployment, these evaluators typically (i) treat each case independently, missing the opportunity to accumulate experience, and (ii) rely on a single fixed prompt for all cases, neglecting the need for sample-specific evaluation criteria. We introduce Learning While Evaluating (LWE), a framework that allows evaluators to improve sequentially at inference time without requiring training or validation sets. LWE maintains an evolving meta-prompt that (i) produces sample-specific evaluation instructions and (ii) refines itself through self-generated feedback. Furthermore, we propose Selective LWE, which updates the meta-prompt only on self-inconsistent cases, focusing computation where it matters most. This selective approach retains the benefits of sequential learning while being far more cost-effective. Across two pairwise comparison benchmarks, Selective LWE outperforms strong baselines, empirically demonstrating that evaluators can improve during sequential testing with a simple selective update, learning most from the cases they struggle with.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
