arXiv 2310.059102024-04-09cs.CLcs.AIcs.LG

SALMON：利用可指令化奖励模型进行自我对齐

SALMON: Self-Alignment with Instructable Reward Models

MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；Language Technologies Institute, CMU（卡内基梅隆大学语言技术研究所）；UMass Amherst（马萨诸塞大学阿默斯特分校）

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

备注：: Previous Title: SALMON: Self-Alignment with Principle-Following Reward Models. Accepted to ICLR 2024. Project page: https://github.com/IBM/SALMON

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SALMON，用合成偏好训练可指令化奖励模型，并依据少量人类原则通过RL对齐LLM；基于LLaMA-2-70b打造的Dromedary-2以6个示例和31条原则超越多个先进系统。

摘要：

在响应示范上进行监督微调（SFT），并结合基于人类反馈的强化学习（RLHF），构成了一种用于对齐基于LLM的AI智能体的强大范式。然而，这种方法的一个显著局限在于其依赖高质量人工标注；由于难以获得一致的响应示范和分布内响应偏好，将其应用于复杂任务颇具挑战。本文提出了一种名为SALMON的新颖方法，能够以最少的人类监督对齐基础语言模型：仅使用少量人类定义的原则，却取得优异性能。我们方法的核心是一个可指令化奖励模型。该模型在合成偏好数据上训练，能够依据任意人类定义的原则生成奖励分数。只需在RL训练阶段调整这些原则，我们就能通过可指令化奖励模型完全控制偏好，进而影响经RL训练的策略模型的行为，并减少对收集在线人类偏好的依赖。将我们的方法应用于LLaMA-2-70b基础语言模型后，我们开发了名为Dromedary-2的AI助手。仅借助6个用于上下文学习的示例和31条人类定义原则，Dromedary-2在多个基准数据集上显著超越了包括LLaMA-2-Chat-70b在内的若干最先进AI系统的性能。我们已开源代码和模型权重，以鼓励进一步研究如何以更高的监督效率、更好的可控性和可扩展的监督来对齐基于LLM的AI智能体。

英文摘要：

Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI agents. However, a significant limitation of such an approach is its dependency on high-quality human annotations, making its application to intricate tasks challenging due to difficulties in obtaining consistent response demonstrations and in-distribution response preferences. This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision, using only a small set of human-defined principles, yet achieving superior performance. Central to our approach is an instructable reward model. Trained on synthetic preference data, this model can generate reward scores based on arbitrary human-defined principles. By merely adjusting these principles during the RL training phase, we gain full control over the preferences with the instructable reward model, subsequently influencing the behavior of the RL-trained policy models, and reducing the reliance on the collection of online human preferences. Applying our method to the LLaMA-2-70b base language model, we developed an AI assistant named Dromedary-2. With only 6 exemplars for in-context learning and 31 human-defined principles, Dromedary-2 significantly surpasses the performance of several state-of-the-art AI systems, including LLaMA-2-Chat-70b, on various benchmark datasets. We have open-sourced the code and model weights to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, improved controllability, and scalable oversight.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。