arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.04283cs.CLcs.AIcs.LG
收藏

Proxy-RLHF:通过代理模型解耦大语言模型的生成与对齐

Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy

University of Science and Technology of China(中国科学技术大学);Deep Space Exploration Laboratory(深空探测实验室);Institute for Advanced Algorithms Research(先进算法研究院);Sangfor Technologies Inc.(深信服科技股份有限公司);Research Institute of China Telecom(中国电信研究院)

Yu Zhu, Chuxiong Sun, Wenfei Yang, Wenqiang Wei, Bo Tang, Tianzhu Zhang, Zhiyu Li, Shifeng Zhang, Feiyu Xiong, Jie Hu, Mingchuan yang

AI总结:

针对现有RLHF方法计算成本高的问题,提出Proxy-RLHF解耦LLM的生成与对齐,通过RL训练精简代理模型监督LLM token生成,仅用1%训练参数就达到相当的对齐效果。

摘要:

基于人类反馈的强化学习(Reinforcement Learning from Human Feedback, RLHF)是确保大语言模型(Large Language Models, LLMs)符合人类价值观的主流方法。然而,现有RLHF方法计算成本高昂,一个主要原因是RLHF同时将生成任务与对齐任务都交由LLM承担。本文提出Proxy-RLHF方法,将LLM的生成过程与对齐过程解耦,以低得多的计算成本实现人类价值观对齐。我们首先设计了一种面向对齐过程的新型马尔可夫决策过程(Markov Decision Process, MDP),并采用强化学习(Reinforcement Learning, RL)训练一个精简的代理模型来监督LLM的token生成,无需修改LLM本身。实验表明,该方法仅使用其他方法1%的训练参数,就能达到相当的对齐水平。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1\% of the training parameters of other methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑