Proximal Policy Optimization (PPO)-based reinforcement learning from human feedback (RLHF) is a widely adopted paradigm for aligning large language models (LLMs) with human preferences. However, its training pipeline suffers from substantial inefficiencies due to sequential multi-model dependencies (e.g., reward model depends on actor outputs) and long-tail response lengths, where a few long responses straggle the stage completion. We present OPPO, a novel, lightweight, and model-agnostic PPO-based RLHF framework that improves training efficiency by overlapping pipeline execution. OPPO introduces two novel techniques: (1) Intra-step overlap, which streams upstream model outputs (e.g., actor model) in right-sized chunks, enabling the downstream model (e.g., reward) to begin prefill while the upstream continues decoding; and (2) Inter-step overlap, which adaptively overcommits a few prompts and defers long generations to future steps, mitigating tail latency without discarding partial work. OPPO integrates easily with existing PPO implementations with a lightweight wrapper. Extensive evaluations show that OPPO accelerates PPO-based RLHF training by $1.8\times$--$2.8\times$ and improves GPU utilization by $1.4\times$--$2.1\times$ without compromising training convergence.
KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF
Computational and Biological Learning Group, Department of Engineering, University of Cambridge, Cambridge, UK（计算生物学学习组，工程系，剑桥大学，剑桥，英国）；Department of Computer Science and Technology, University of Cambridge, Cambridge, UK（计算机科学与技术系，剑桥大学，剑桥，英国）；Statistics Laboratory, Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, UK（统计实验室，纯粹数学与数学统计系，剑桥大学，英国）
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））；China Shenzhen International Center For Industrial and Applied Mathematics（中国深圳国际工业与应用数学中心）
Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
School of Mathematics and Statistics（数学与统计学院）；Beijing Jiaotong University（北京交通大学）；University of Science and Technology of China（中国科学技术大学）；School of information（信息学院）；School of Management（管理学院）；Renmin University of China（中国人民大学）；Fudan University（复旦大学）；Academy of Mathematics and Systems Science（数学与系统科学研究院）
RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders
Northeastern University（东北大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Northwestern University（西北大学）；University of Nebraska - Lincoln（内布拉斯加大学林肯分校）；Columbia University（哥伦比亚大学）
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Salesforce AI Research（Salesforce AI研究院）；The Hong Kong University of Science and Technology（香港科技大学）
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Qwen LLM Application Team, Alibaba（阿里巴巴Qwen大模型应用团队）；Stanford University（斯坦福大学）
Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Department of Computer Science（计算机科学系）；Engineering, The Ohio State University（工程系，俄亥俄州立大学）
