arXiv 2406.068742024-11-29cs.AIcs.HCcs.RO

从示范与偏好中联合学习奖励和策略可改善对齐

Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment

University of Minnesota（明尼苏达大学）；The Ohio State University（俄亥俄州立大学）；Texas A&M University（德克萨斯农工大学）

Chenliang Li, Siliang Zeng, Zeyi Liao, Jiaxiang Li, Dongyeop Kang, Alfredo Garcia, Mingyi Hong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对RLHF等多阶段对齐方法的数据利用不足与分布失配问题，提出单阶段AIHF框架，联合利用示范和偏好训练奖励模型与策略，并在LLM和MuJoCo任务中显著优于RLHF、DPO。

摘要：

对齐人类偏好与价值是构建当代基础模型和具身AI的重要要求。然而，诸如基于人类反馈的强化学习（RLHF）等流行方法将任务分解为连续阶段，例如监督微调（SFT）、奖励建模（RM）和强化学习（RL），每个阶段执行一项特定学习任务。这种顺序方法导致严重问题，例如数据严重利用不足，以及所学奖励模型与生成策略之间的分布不匹配，最终造成较差的对齐性能。我们开发了一种名为 Alignment with Integrated Human Feedback（AIHF，整合人类反馈的对齐）的单阶段方法，能够整合人类偏好和示范来训练奖励模型与策略。所提出的方法支持一组高效算法，这些算法可以轻松归约为并利用 RLHF 和 Directly Policy Optimization（DPO，直接策略优化）等流行对齐算法，并且只需要对现有对齐流程进行微小改动。我们通过涉及LLM对齐问题和MuJoCo中机器人控制问题的大量实验，证明了所提解决方案的效率。我们观察到，所提解决方案以大幅优势优于 RLHF 和 DPO 等现有对齐算法，尤其是在高质量偏好数据数量相对有限时。

英文摘要：

Aligning human preference and value is an important requirement for building contemporary foundation models and embodied AI. However, popular approaches such as reinforcement learning with human feedback (RLHF) break down the task into successive stages, such as supervised fine-tuning (SFT), reward modeling (RM), and reinforcement learning (RL), each performing one specific learning task. Such a sequential approach results in serious issues such as significant under-utilization of data and distribution mismatch between the learned reward model and generated policy, which eventually lead to poor alignment performance. We develop a single stage approach named Alignment with Integrated Human Feedback (AIHF), capable of integrating both human preference and demonstration to train reward models and the policy. The proposed approach admits a suite of efficient algorithms, which can easily reduce to, and leverage, popular alignment algorithms such as RLHF and Directly Policy Optimization (DPO), and only requires minor changes to the existing alignment pipelines. We demonstrate the efficiency of the proposed solutions with extensive experiments involving alignment problems in LLMs and robotic control problems in MuJoCo. We observe that the proposed solutions outperform the existing alignment algorithms such as RLHF and DPO by large margins, especially when the amount of high-quality preference data is relatively limited.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。