arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2603.08035cs.AIcs.LG
收藏

CDRRM:基于对比的评分标准生成用于可靠和可解释的奖励建模

CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling

University of Science and Technology of China(科学技术大学);Peking University(北京大学);BeiHang University(北航大学)

Dengcan Liu, Fengkai Yang, Xiaohan Wang, Shurui Yan, Jiajun Chai, Jiahao Li, Yikun Ban, Zhendong Mao, Wei Lin, Guojun Yin

AI总结:

CDRRM通过对比驱动的评分标准生成方法,实现可靠且可解释的奖励建模,有效缓解评估偏见并提升数据效率。

摘要:

奖励建模对于对齐大语言模型(LLMs)与人类偏好至关重要,但传统奖励模型存在可解释性差和过度依赖昂贵专家注释的问题。尽管最近基于评分标准的方法增强了评估透明度,但它们缺乏系统质量控制,导致评分标准嘈杂且冗余,未能缓解LLM评估者中持续存在的偏见(例如冗长性、位置),并造成了可扩展性与可靠性之间的权衡。为了解决这些限制,我们提出了CDRRM(基于对比的评分标准奖励模型),该框架基于一种新颖的对比后综合范式,用于高质量评分标准生成和引导偏好判断。CDRRM首先在偏好对上进行多维对比分析,以识别因果区分因素,然后将这些洞察综合成紧凑、上下文感知的评分标准,以指导偏好判断。在三个权威基准(RewardBench、RMBench、RMB)上的广泛实验表明,CDRRM在不同领域均实现了最先进的性能,并有效缓解了上述评估偏见。值得注意的是,我们的方法在数据效率方面表现突出:仅在3k高质量样本上训练评分标准生成器,即可使冻结的预训练判断模型超越完全微调的基线。这项工作为奖励建模提供了一条可扩展、可解释且数据高效的路径。

英文摘要:

Reward modeling is essential for aligning Large Language Models(LLMs) with human preferences, yet conventional reward models suffer from poor interpretability and heavy reliance on costly expert annotations. While recent rubric-based approaches enhance evaluation transparency, they lack systematic quality control, yielding noisy and redundant criteria, failing to mitigate persistent biases (e.g., verbosity, position) in LLM evaluators, and creating a scalability-reliability trade-off. To address these limitations, we propose CDRRM (Contrast-Driven Rubric Reward Model), a framework built on a novel Contrast-then-Synthesis paradigm for high-quality rubric generation and guided preference judgment. CDRRM first conducts multi-dimensional contrastive profiling on preference pairs to identify causal discriminative factors, then synthesizes these insights into compact, context-aware rubrics to guide preference judg- ments. Extensive experiments on three authoritative benchmarks (RewardBench, RMBench, RMB) demonstrate that CDRRM achieves state-of-the-art performance across diverse domains and effectively mitigates aforementioned evaluation biases. Notably, our approach delivers exceptional data efficiency: training the rubric generator on only 3k high-quality samples empowers a frozen pre-trained judge model to outperform fully fine-tuned baselines. This work offers a scalable, interpretable, and data-efficient path for reward modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑