arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01470cs.CL
收藏

DogeRM:通过模型融合为奖励模型配备领域知识

DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging

National Taiwan University(国立台湾大学)

Tzu-Han Lin, Chen-An Li, Hung-yi Lee, Yun-Nung Chen

AI总结:

针对RLHF中领域偏好数据标注成本高的问题,提出DogeRM框架,通过模型融合将领域知识整合到通用奖励模型中,在多基准上提升性能,为模型对齐提供新思路。

摘要:

基于人类反馈的强化学习(RLHF)是使大语言模型(LLM)与预期行为对齐的主流策略,奖励建模是RLHF的关键步骤。然而,收集用于训练奖励模型的成对偏好数据往往成本高、耗时长,针对需要专家标注的特定领域偏好数据更是如此。为解决这一问题,我们提出了领域知识融合奖励模型(DogeRM),这是一种通过模型融合将特定领域知识整合到通用奖励模型中的新型框架。实验表明,DogeRM在多个基准上提升了性能,我们还通过详细分析展示了模型融合的效果,凸显了其助力模型对齐的巨大潜力。

英文摘要:

Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the \textbf{Do}main knowled\textbf{ge} merged \textbf{R}eward \textbf{M}odel (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑