分组边缘化自奖励强化学习驱动零标签自我进化
Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving
- Shanghai Jiao Tong University（上海交通大学）
- Tencent（腾讯）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对自奖励强化学习中分组上下文随机性导致奖励信号缺失的问题，提出分组边缘化优势估计（GMAE），聚合跨上下文奖励实现以估计期望优势，在八个基准和四个模型上验证了性能与泛化。
AI中文摘要：
自奖励强化学习（Self-rewarding reinforcement learning）使大型语言模型（LLMs）无需人类标签即可自我进化。现有的基于集成的方法从 rollout 分组中构建奖励参考并据此分配奖励。然而，一个响应的奖励表示也依赖于其随机采样的分组上下文，即同组中的其他响应。仅使用一种分组上下文实现可能会错过期望的奖励信号，并为策略优化提供不可靠的指导。为解决此问题，我们提出了分组边缘化优势估计（Group-Marginalized Advantage Estimation, GMAE），该方法将跨可能上下文的奖励实现聚合为响应级分布，并估计期望优势。在八个基准和四个基础模型上的实验表明，该方法具有强大的性能和跨领域泛化能力。GMAE 还表现出稳定的学习、低额外成本，以及跨训练数据集和强化学习骨干的良好适用性。
英文摘要：
Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response's reward representation also depends on its randomly sampled group context, i.e., the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose Group-Marginalized Advantage Estimation (GMAE), which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones.
