是时候摒弃LLM-as-a-Judge了：程序是评估的未来
Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation
AI总结：
针对LLM-as-a-Judge评估方式成本高、可靠性差、有偏差等问题，研究提出PAJAMA框架，用LLM合成可执行裁判程序实现评估，成本更低、一致性与表现更优。
摘要：
大语言模型（LLM）被广泛用于评估LLM生成内容与回复的质量，但这带来了显著挑战：高昂的API成本、不确定的可靠性、不灵活的流水线以及固有偏差。为解决这些问题，我们提出PAJAMA（Program-As-a-Judge for Automated Model Assessment，即基于程序作为裁判的自动化模型评估）——一种全新替代方案，它利用LLM合成可执行的裁判程序，而非直接对回复打分。这些合成的程序可存储并在本地运行，成本低数个数量级，同时提供可解释、可审计的裁判逻辑，且易于适配调整。基于程序的裁判可缓解偏差，与基于Qwen2.5-14B的LLM-as-a-Judge相比，判断一致性提升15.83%，偏差回复平均减少23.7%。当程序裁判被蒸馏到模型中时，PAJAMA在RewardBench的高难度CHAT-HARD子集上表现优于LLM-as-a-Judge，在Prometheus上的指标表现优2.19%，在JudgeLM数据集上优8.67%，而成本仅为其三到四个数量级。
英文摘要：
Large language models (LLMs) are widely used to evaluate the quality of LLM generations and responses, but this leads to significant challenges: high API costs, uncertain reliability, inflexible pipelines, and inherent biases. To address these, we introduce PAJAMA (Program-As-a-Judge for Automated Model Assessment), a new alternative that uses LLMs to synthesize executable judging programs instead of directly scoring responses. These synthesized programs can be stored and run locally, costing orders of magnitude less while providing interpretable, and auditable judging logic that can be easily adapted. Program-based judges mitigate biases, improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average compared to a Qwen2.5-14B-based LLM-as-a-judge. When program judgments are distilled into a model, PAJAMA outperforms LLM-as-a-judge on the challenging CHAT-HARD subset of RewardBench, outperforming metrics by 2.19% on Prometheus and 8.67% on the JudgeLM dataset, all at three orders of magnitude lower cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
