近来的文献将近端策略优化（PPO）定位为强化学习从人类反馈（RLHF）部分的标准方法。PPO在经验上表现良好，但其动机具有启发式性质，并且在LM-RLHF中处理KL散度约束的方式是随意的，导致奖励震荡、熵崩溃、价值函数漂移和突然的策略发散，需要频繁重启和大量的超参数调整。在本文中，我们开发了一种新的纯在线策略演员-评论家RL方法，用于LM-RLHF设置。我们提出了SAFE（具有熵感知控制的稳定对齐微调），一种新的RLHF算法，结合了双软最小批评器用于悲观价值估计，以及一个新的多层稳定框架，结合熵门控KL调节和PID控制的自适应阈值。与标准PPO的对称KL惩罚不同，SAFE区分高熵探索和低熵模式崩溃，并根据奖励速度动态调整惩罚。在3B参数模型上的实验表明，SAFE在训练平均奖励方面比PPO高5.15%（0.725 vs 0.689），奖励崩溃可忽略，KL控制优于PPO。我们的方法增加了最小的计算开销，并提供了一个可解释、抗崩溃的RLHF框架，该框架在保持积极学习速度的同时，确保了适合生产部署的稳定长周期优化。代码可在https://github.com/ryyzn9/SAFE上获得。
英文摘要：
Proximal Policy Optimization (PPO) has been positioned by recent literature as the canonical method for the RL part of Reinforcement Learning from Human Feedback (RLHF). PPO performs well empirically but has a heuristic motivation and handles the KL-divergence constraint used in LM-RLHF in an ad-hoc manner and suffers form reward oscillations, entropy collapse, value function drift, and sudden policy divergence that require frequent restarts and extensive hyperparameter tuning. In this paper, we develop a new pure on policy actor-critic RL method for the LM-RLHF setting. We present SAFE (Stable Alignment Finetuning with Entropy-aware control),a novel RLHF algorithm that combines a Double Soft-Min Critic for pessimistic value estimation with a new multi-layer stabilization framework combining entropy-gated KL regulation, and PID-controlled adaptive thresholds. Unlike standard PPO's symmetric KL penalties, SAFE distinguishes high-entropy exploration from low-entropy mode collapse and adjusts penalties dynamically based on reward velocity. Experiments on a 3B parameter model show SAFE achieves +5.15\% training-average reward than PPO (0.725 vs 0.689), negligible reward crashes, and superior KL control than ppo . Our method adds minimal computational overhead and provides an interpretable, crash-resistant RLHF framework that maintains aggressive learning speed while ensuring stable long-horizon optimization suitable for production deployment. Code is available at https://github.com/ryyzn9/SAFE
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
National University of Defense Technology（国防科技大学）；State Key Laboratory of Complex & Critical Software Environment（复杂与关键软件环境国家重点实验室）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment
Department of Electrical and Computer Engineering（电气与计算机工程系）；New Jersey Institute of Technology（新泽西理工学院）；Department of Computer Engineering（计算机工程系）；Heritage Institute of Technology（遗产理工学院）
Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
School of Electrical, Computer and Energy Engineering（电气、计算机与能源工程学院）；Arizona State University（亚利桑那州立大学）；School of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；Washington State University（华盛顿州立大学）；Department of Computer Science（计算机科学系）；University of Houston（休斯顿大学）
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
MIT（麻省理工学院）；Harvard University（哈佛大学）；Columbia University（哥伦比亚大学）；Cornell Tech（康奈尔科技校区）；Apollo Research（阿波罗研究院）；ETH Zurich（苏黎世联邦理工学院）；UC Berkeley（加州大学伯克利分校）；University of Sussex（萨塞克斯大学）；EffiSciences；University of Cambridge（剑桥大学）；New York University（纽约大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of Southern California（南加州大学）；Stanford University（斯坦福大学）
Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
School of Mathematics and Statistics（数学与统计学院）；Beijing Jiaotong University（北京交通大学）；University of Science and Technology of China（中国科学技术大学）；School of information（信息学院）；School of Management（管理学院）；Renmin University of China（中国人民大学）；Fudan University（复旦大学）；Academy of Mathematics and Systems Science（数学与系统科学研究院）
WST: Weak-to-Strong Knowledge Transfer via Reinforcement Learning
Wharton AI & Analytics Initiative（沃顿人工智能与分析倡议）；University of Pennsylvania（宾夕法尼亚大学）；Department of Computer and Information Science（计算机与信息科学系）；Department of Physics and Astronomy（物理学与天文学系）
HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning
Sony AI（索尼人工智能公司）；Graduate Institute of Communication Engineering, National Taiwan University（台湾大学通信工程研究所）；University of Southern California（南加利福尼亚大学）；Stanford University（斯坦福大学）
Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma
Berkeley AI Safety Initiative (BASIS), University of California, Berkeley（伯克利人工智能安全倡议（BASIS），加州大学伯克利分校）；AWS Generative AI Innovation Center, Amazon Web Services（亚马逊网络服务生成式人工智能创新中心）；Meta AI；Stanford University（斯坦福大学）；Northeastern University, Seattle, WA, USA（东北大学，西雅图，华盛顿州，美国）
