arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.12464cs.CL
收藏

探究偏好表示:奖励模型的多维度评估与分析方法

Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Meituan Inc.(美团公司);NiuTrans Research(牛译研所);CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS(中国科学院行为科学重点实验室);Kunming University of Science and Technology(昆明理工大学)

Chenglong Wang, Yifu Huo, Yang Gan, Yongyu Mu, Qiaozhi He, Murun Yang, Bei Li, Chunliang Zhang, Tongran Liu, Anxiang Ma, Zhengtao Yu, Jingbo Zhu, Tong Xiao

AI总结:

针对以往奖励模型评估缺乏维度性能信息的问题,本研究构建多维度奖励模型基准MRMBench并提出推理时探究方法,验证其与LLMs对齐性能强相关,揭示奖励模型多维度偏好捕捉不足问题及多目标优化潜力,提升预测置信度与对齐效果。

摘要:

以往方法通过在固定的成对排序测试集上测试奖励模型进行评估,但通常不提供各偏好维度的性能信息。本研究通过探究偏好表示来解决奖励模型的评估挑战。为验证该评估方法的有效性,我们构建了多维度奖励模型基准(MRMBench),包含针对不同偏好维度的6项探究任务,旨在青睐并鼓励能更好捕捉多维度偏好的奖励模型。此外,我们引入推理时探究分析方法,识别奖励预测过程中使用的维度并增强其可解释性。通过大量实验,我们发现MRMBench与大型语言模型(LLMs)的对齐性能强相关,成为开发先进奖励模型的可靠参考;分析MRMBench评估结果显示,奖励模型常难以捕捉多维度偏好,凸显多目标优化在奖励建模中的潜力;同时,推理时探究方法为评估奖励预测置信度提供可靠指标,最终提升LLMs的对齐效果。

英文摘要:

Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with the alignment performance of large language models (LLMs), making it a reliable reference for developing advanced reward models. Our analysis of MRMBench evaluation results reveals that reward models often struggle to capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Additionally, our findings show that the proposed inference-time probing method offers a reliable metric for assessing the confidence of reward predictions, which ultimately improves the alignment of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑