COPR：通过最优策略正则化进行人类偏好持续学习
COPR: Continual Learning Human Preference through Optimal Policy Regularization
AI总结：
针对RLHF模型需反复重训练以适应新人类偏好的资源与隐私问题，提出COPR方法，通过绕过配分函数计算最优策略分布并基于历史分布正则化当前策略以缓解灾难性遗忘，实现无需复杂强化学习的持续对齐。
摘要：
人类反馈强化学习（RLHF）技术是改进预训练语言模型（LM）的常用方法，能增强其符合人类偏好的能力。然而，当前基于RLHF的LM每次引入新查询或反馈时都需要全面重新训练，由于人类偏好在不同领域或任务间存在差异，这成为一项挑战。重新训练LM在许多现实场景中面临实际困难，因为需要大量时间和计算资源，并伴随数据隐私问题。为解决这一局限，我们提出一种名为持续最优策略正则化（COPR）的新方法：通过绕过配分函数计算最优策略分布，并基于历史最优分布对当前策略进行正则化，以缓解灾难性遗忘（CF）。COPR仅需单一学习阶段，无需复杂强化学习。重要的是，它通过维护类似奖励模型的评分模块，保留了RLHF从无标签数据学习的能力，使其能灵活地持续学习而无需人类反馈。实验结果表明，在增量任务和领域上持续对齐人类偏好方面，COPR优于强大的持续学习（CL）基线方法。
英文摘要：
The technique of Reinforcement Learning from Human Feedback (RLHF) is a commonly employed method to improve pre-trained Language Models (LM), enhancing their ability to conform to human preferences. Nevertheless, the current RLHF-based LMs necessitate full retraining each time novel queries or feedback are introduced, which becomes a challenging task because human preferences can vary between different domains or tasks. Retraining LMs poses practical difficulties in many real-world situations due to the significant time and computational resources required, along with concerns related to data privacy. To address this limitation, we propose a new method called Continual Optimal Policy Regularization (COPR), in which we compute the distribution of optimal policy bypassing the partition function and then regularize the current policy based on the historically optimal distribution to mitigate Catastrophic Forgetting (CF). COPR involves a single learning phase and doesn't necessitate complex reinforcement learning. Importantly, it shares the capability with RLHF to learn from unlabeled data by maintaining a scoring module, similar to reward model, making it flexible for continually learning without human feedback. Our experimental results show that COPR outperforms strong Continuous Learning (CL) baselines when it comes to consistently aligning with human preferences on incremental tasks and domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
