修正的内生奖励
Triviality Corrected Endogenous Reward
AI总结:
本文提出TCER,通过相对信息增益和概率修正机制解决开放性写作中内生奖励的平凡偏差问题,提升生成多样性与内容质量。
摘要:
开放性文本生成的强化学习受限于可验证奖励的缺乏,需依赖标注数据或强大闭源模型。受近期无监督强化学习用于数学推理的启发,我们探讨内生奖励原理是否适用于开放性写作任务。发现直接应用置信度奖励导致平凡偏差:策略趋向高概率输出,降低多样性与有意义内容。我们提出TCER(修正的内生奖励),通过专家策略与通用策略间的相对信息增益奖励,结合概率依赖修正机制解决此偏差。在多个写作基准和模型架构上,TCER实现一致改进,无需外部监督。此外,TCER有效迁移至数学推理,验证了方法在不同生成任务中的通用性。
英文摘要:
Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。