重新思考评估标准生成以改进LLM评判与开放任务的奖励建模
Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
AI总结：
RRD通过递归分解-过滤循环优化评估标准，提升LLM评判准确性和奖励建模效果。
摘要：
最近，评估标准已被用于指导LLM评判者捕捉主观、细微的多维人类偏好，并已扩展到用于强化微调（RFT）的奖励信号。然而，评估标准的生成仍然难以控制：评估标准通常缺乏覆盖范围，混淆维度，偏好方向不一致，并包含冗余或高度相关的标准，从而降低评判准确性并产生次优奖励。我们提出了RRD，一种基于递归分解-过滤循环的评估标准优化框架。RRD将粗略的评估标准分解为细粒度、判别性的标准，扩大覆盖范围的同时增强响应之间的分离度。一个互补的过滤机制移除不一致和冗余的评估标准，一个相关性感知的加权方案防止高度相关标准的过度表示，从而产生信息丰富、全面且非冗余的评估标准集。经验上，RRD在评估和训练方面均实现了显著的提升：它在JudgeBench和PPE上提高了GPT-4o和Llama3.1-405B评判者的偏好判断准确性，实现了所有设置中的最佳表现，最高在JudgeBench上提高了+17.7分。当用于在WildChat上用于RFT时，它产生了显著更强且更稳定的训练信号，使奖励提高了最高160%（Qwen3-4B）和60%（Llama3.1-8B），相较于先前的评估标准基线提高了10-20%，并且这些收益转移到了HealthBench-Hard和BiGGen Bench。总体而言，RRD确立了递归的评估标准优化作为LLM评判和开放领域奖励建模的可扩展且可解释的基础。
英文摘要：
Recently, rubrics have been used to guide LLM judges in capturing subjective, nuanced, multi-dimensional human preferences, and have been extended from evaluation to reward signals for reinforcement fine-tuning (RFT). However, rubric generation remains hard to control: rubrics often lack coverage, conflate dimensions, misalign preference direction, and contain redundant or highly correlated criteria, degrading judge accuracy and producing suboptimal rewards during RFT. We propose RRD, a principled framework for rubric refinement built on a recursive decompose-filter cycle. RRD decomposes coarse rubrics into fine-grained, discriminative criteria, expanding coverage while sharpening separation between responses. A complementary filtering mechanism removes misaligned and redundant rubrics, and a correlation-aware weighting scheme prevents over-representing highly correlated criteria, yielding rubric sets that are informative, comprehensive, and non-redundant. Empirically, RRD delivers large, consistent gains across both evaluation and training: it improves preference-judgment accuracy on JudgeBench and PPE for both GPT-4o and Llama3.1-405B judges, achieving top performance in all settings with up to +17.7 points on JudgeBench. When used as the reward source for RFT on WildChat, it yields substantially stronger and more stable learning signals, boosting reward by up to 160% (Qwen3-4B) and 60% (Llama3.1-8B) versus 10-20% for prior rubric baselines, with gains that transfer to HealthBench-Hard and BiGGen Bench. Overall, RRD establishes recursive rubric refinement as a scalable and interpretable foundation for LLM judging and reward modeling in open-ended domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
