Reinforcement Learning from Human Feedback (RLHF) has advanced alignment capabilities significantly but remains hindered by two core challenges: \textbf{reward hacking} and \textbf{stable optimization}. Current solutions independently address these issues through separate regularization strategies, specifically a KL-divergence penalty against a supervised fine-tuned model ($π_0$) to mitigate reward hacking, and policy ratio clipping towards the current policy ($π_t$) to promote stable alignment. However, the implicit trade-off arising from simultaneously regularizing towards both $π_0$ and $π_t$ remains under-explored. In this paper, we introduce a unified regularization approach that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates. Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity. Extensive experiments across diverse benchmarks validate that our method consistently outperforms RLHF and online preference learning methods, achieving enhanced alignment performance and stability.
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
National University of Defense Technology（国防科技大学）；State Key Laboratory of Complex & Critical Software Environment（复杂与关键软件环境国家重点实验室）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
University of Pennsylvania（宾夕法尼亚大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National University of Singapore（新加坡国立大学）；Peking University（北京大学）；Joint corresponding authors（联合通讯作者）
$f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
Department of Electrical and Computer Engineering, University of Virginia, Virginia, United States（弗吉尼亚大学电气与计算机工程系）；Princeton Language and Intelligence, Princeton University, New Jersey, United States（普林斯顿大学语言与智能）
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
University of California, Berkeley（加州大学伯克利分校）；Center for Advanced Intelligence Project, RIKEN（RIKEN先进智能项目中心）；The Institute of Statistical Mathematics and the Graduate University for Advanced Studies（统计数理研究所与综合研究大学院大学）；Tohoku University（东北大学）
Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程技术研究中心，教育部）；Kuaishou Technology（快手科技）；University of Chinese Academy of Sciences（中国科学院大学）
École Polytechnique（巴黎综合理工学院）；HSE University（高等经济大学）；Google DeepMind（谷歌DeepMind）；Mohamed Bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；IDEMIA（Morpho公司（IDEMIA））；ENS Lyon（里昂高等师范学院）
Technische Universität Darmstadt（达姆施塔特工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Monte-Carlo planning and Reinforcement Learning (RL) are essential to sequential decision making. The recent AlphaGo and AlphaZero algorithms have shown how to successfully combine these two paradigms in order to solve large scale sequential decision problems. These methodologies exploit a variant of the well-known UCT algorithm to trade off exploitation of good actions and exploration of unvisited states, but their empirical success comes at the cost of poor sample-efficiency and high computation time. In this paper, we overcome these limitations by considering convex regularization in Monte-…
Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；The State Key Lab of Brain-Machine Intelligence, Zhejiang University（浙江大学脑机智能国家重点实验室）；LLM Safety Centre, Beijing Academy of Artificial Intelligence（北京人工智能研究院大语言模型安全中心）；Center for AI Safety and Governance, Peking University（北京大学人工智能安全与治理中心）
Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
Shanghai Qi Zhi Institute（上海期智研究院）；Tsinghua University（清华大学）；Shanghai AI Lab（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
VinUniversity（文大学）；VinUni-Illinois Smart Health Center（VinUni-伊利诺伊智能健康中心）；University of Stuttgart（斯图加特大学）；University of California Los Angeles (UCLA)（加州大学洛杉矶分校）；International University - VNUHCM（国际大学 - VNUHCM）；University of Science - VNUHCM（科学大学 - VNUHCM）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Oldenburg University（奥尔登堡大学）；Max Planck Research School for Intelligent Systems (IMPRS-IS)（马克斯·普朗克智能系统研究学校）
Many previous proposals for adversarial training of deep neural nets have included di- rectly modifying the gradient, training on a mix of original and adversarial examples, using contractive penalties, and approximately optimizing constrained adversarial ob- jective functions. In this paper, we show these proposals are actually all instances of optimizing a general, regularized objective we call DataGrad. Our proposed DataGrad framework, which can be viewed as a deep extension of the layerwise contractive au- toencoder penalty, cleanly simplifies prior work and easily allows extensions such a…
COPR: Continual Learning Human Preference through Optimal Policy Regularization
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；King’s College London（伦敦国王学院）；National University of Defense Technology（国防科技大学）
