arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.12845cs.LGcs.CL
收藏

通过多目标奖励建模与混合专家实现可解释偏好

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Wisconsin–Madison(威斯康星大学麦迪逊分校)

Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang

AI总结:

针对传统奖励模型黑箱不可解释的问题,提出结合多维度绝对评分多目标奖励模型ArmoRM与混合专家门控策略的两阶段方法,在RewardBench上取得SOTA,性能超GPT-4评判法且接近大参数量奖励模型。

摘要:

基于人类反馈的强化学习(RLHF)已成为使大语言模型(LLM)与人类偏好对齐的主要方法。RLHF流程通常首先使用人类偏好数据训练奖励模型(RM)。传统奖励模型基于同一用户请求的成对响应进行训练,通过相对评分表明人类更偏好哪一响应,训练后的奖励模型充当人类偏好的代理。然而,由于奖励模型的黑箱特性,其输出缺乏可解释性,人类无法直观理解奖励模型为何认为某一响应是好的或不好的。鉴于奖励模型作为人类偏好代理的角色,我们认为它们应具备人类可解释性,以确保其内部决策过程与人类偏好一致,并防止LLM对齐中的奖励黑客问题。为构建具备可解释偏好的奖励模型,我们提出一种两阶段方法:i)利用多维度绝对评分数据训练绝对评分多目标奖励模型(ArmoRM,Absolute-Rating Multi-Objective Reward Model),每个维度对应一个人类可解释的目标(例如诚实性、详细程度、安全性);ii)采用带门控网络的混合专家(MoE,Mixture-of-Experts)策略,门控网络可根据上下文自动选择最合适的奖励目标。我们使用Llama-3 8B高效训练了ArmoRM,并在ArmoRM之上搭建了由浅层MLP构成的门控网络。我们训练的模型ArmoRM-Llama3-8B在RewardBench(一个评估语言建模奖励模型的基准)上取得了当前最优性能。值得注意的是,我们的模型性能显著超过了以GPT-4作为评判的LLM-as-a-judge方法,且性能接近规模大得多的Nemotron-4 340B奖励模型。

英文摘要:

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, we believe they should be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we propose a two-stage approach: i) train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art performance on RewardBench, a benchmark evaluating RMs for language modeling. Notably, the performance of our model surpasses the LLM-as-a-judge method with GPT-4 judges by a margin, and approaches the performance of the much larger Nemotron-4 340B reward model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑