发表机构
Microsoft Research; Tsinghua University; Peking University（微软研究院; 清华大学; 北京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究不可验证任务，提出体验式学习（EL），将LLM反馈模型从评判转为教练，通过提炼体验知识提供密集监督，在开放式任务上表现优于基于规则的RL，泛化性好且减轻奖励作弊。
AI 中文摘要
在开放式任务上的强化学习（RL）将基于大语言模型（LLM）的基于规则的评估压缩为标量奖励，丢弃了丰富的文本反馈，并将具有不同质量配置文件的响应混为一谈。我们提出了体验式学习（EL），它将反馈模型从作为评判的LLM重新用作作为教练的LLM。教练将其对每个策略响应的评估提炼为可转移的体验知识，该知识为教师模型提供条件，并通过策略上下文提炼被策略内化。与标量奖励相比，这个更高带宽的反馈通道提供了密集监督，并保留了高质量响应之间的细粒度偏好。在两个策略家族中，有来自策略本身或专有模型的反馈，EL在留出的和未见的开放式任务上始终优于基于规则的RL。值得注意的是，EL在训练分布之外具有更好的泛化能力，并减轻了奖励作弊。这些发现将体验知识确立为用于不可验证任务的训练后更丰富、更可泛化的学习信号。
英文摘要
Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.
