用于人类反馈强化学习的元学习奖励塑形
Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
浏览论文内容
中文总结 AI 辅助
MeRLa是元学习的任务感知奖励塑形框架，在RLHF训练前通过辅助任务元学习塑形函数，可提升LLaMA-3-8B在多基准上的对齐性能，降低训练不稳定性。
中文摘要 AI 辅助
人类反馈强化学习（RLHF）是使大型语言模型对齐人类偏好的标准方法，但其质量受限于静态、与任务无关的奖励模型，这种不匹配会导致学习信号稀疏和对齐效果不佳。我们提出MeRLa（元学习奖励塑形框架），这是一个在RLHF训练前通过辅助任务元学习任务感知塑形函数Φ(x,y;φ)的原则性框架，学习到的塑形会生成复合奖励，在保持策略最优性的同时提供特定任务的学习信号。我们的元目标结合了任务判别、熵正则化和基于势能的守恒以实现稳定收敛，为策略不变性提供了理论保证，分析了表示漂移敏感性，并正式解决了熵最大化带来的激励对齐问题。在LLaMA-3-8B上针对四个基准的实验显示，MeRLa相比PPO、DPO、GRPO和DAPO取得了一致的性能提升，在AlpacaEval 2.0上达到90.8%的长度控制胜率，在MT-Bench上得分9.14，训练不稳定性降低41%，且MeRLa与基于过程和基于规则的增强奖励结合时仍能保持其优势。
英文摘要
Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function $Φ(x,y;ϕ)$ across auxiliary tasks before RLHF training. The learned shaping produces a composite reward that preserves policy optimality while providing task-specific learning signals. Our meta-objective combines task discrimination, entropy regularization, and potential-based conservation for stable convergence. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization. Experiments on LLaMA-3-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and a score of 9.14 on MT-Bench, with 41% less training instability. MeRLa retains its benefits when combined with process-based and rubric-based enhanced rewards.
