LLM裁判可靠性检测工具：检验LLM裁判的可靠性
Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
AI总结：
本研究提出了一种开源工具，用于评估LLM裁判在不同基准中的可靠性，发现模型和扰动类型对性能有显著影响。
摘要：
我们提出了Judge Reliability Harness，一个开源库，用于构建验证套件，以测试LLM裁判的可靠性。随着基于LLM的评分在AI基准中被广泛部署，需要更多的工具来高效评估这些方法的可靠性。给定一个基准数据集和一个LLM裁判配置，该工具生成可靠性测试，评估自由响应和代理任务格式中的二元判断准确性和顺序评分性能。我们评估了四个最先进的裁判器在四个涵盖安全、说服力、滥用和代理行为的基准中的表现，发现模型和扰动类型之间存在有意义的性能差异，突显了提高LLM裁判鲁棒性的机会。我们评估的任何裁判器在使用我们的工具时都不是在所有基准中都可靠。例如，我们对裁判器的初步实验揭示了由于简单的文本格式变化、改写、 verbosity变化以及翻转LLM生成响应的地面真实标签而导致的一致性问题。此工具的代码可在：https://github.com/RANDCorporation/judge-reliability-harness获取。
英文摘要：
We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
