LLM-as-a-Judge 能否可靠地验证智能体场景中的评分标准？
Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
AI总结：
针对 LLM 作为裁判在智能体场景中验证评分标准的可靠性问题，构建首个基准 RuVerBench，评估多种前沿模型，发现即使最强模型也存在显著噪声，并分析了提示设计、批处理和多数投票等策略的影响。
摘要：
基于评分标准的评分已成为模型评估中广泛使用的范式，通常采用 LLM-as-a-Judge (LaaJ) 进行评分。然而，LaaJ 用于评分标准的可靠性仍未得到充分探索。这一问题在智能体场景中尤为突出，因为长而复杂的输出进一步挑战了可靠评分。为解决此问题，我们对 LaaJ 在评分标准验证中的可靠性进行了系统的元评估。我们引入了 RuVerBench，这是首个评估 LaaJ 在智能体场景中评分标准验证可靠性的基准。RuVerBench 涵盖两个常见的智能体领域：深度研究和智能体编码，包含 2,458 个实例，每个实例包含模型生成的输出、评分标准以及人工标注的标签，指示输出是否满足评分标准。使用 RuVerBench，我们评估了多个前沿 LLM，发现即使是最先进的模型也取得了强劲的性能，但仍然表现出显著的噪声。我们进一步分析了关键 LaaJ 策略（包括提示设计、批处理和多数投票）对评分标准验证的影响。我们发现，较弱的模型对提示变化更敏感，批处理验证在准确性和效率之间存在权衡，而多数投票产生了有效但递减的回报。我们已发布数据集和代码以促进未来研究：https://github.com/THU-KEG/RuVerBench。
英文摘要：
Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric scoring remains underexplored. This concern is especially pronounced in agentic scenarios, where long, complex outputs further challenge reliable scoring. To address this, we conduct a systematic meta-evaluation of LaaJ reliability for rubric verification. We introduce RuVerBench, the first benchmark for assessing LaaJ reliability in rubric verification for agentic scenarios. RuVerBench covers two prevalent agentic domains, deep research and agentic coding, with 2,458 instances, each containing a model-generated output, a rubric, and a human-annotated label indicating whether the output satisfies the rubric. Using RuVerBench, we evaluate numerous frontier LLMs and find that even the most advanced models achieve strong performance but still exhibit substantial noise. We further analyze the impact of key LaaJ strategies, including prompt design, batching, and majority voting, on rubric verification. We find that weaker models are more sensitive to prompt variations, batched verification presents a trade-off between accuracy and efficiency, and majority voting yields effective but diminishing returns. We have released our dataset and code to facilitate future research: https://github.com/THU-KEG/RuVerBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
