arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.16259cs.LGcs.AI
收藏

超越分布细化:任务奖励的重要性

Beyond Distribution Sharpening: The Importance of Task Rewards

Mila;Université de Montréal(蒙特利尔大学)

Sarthak Mittal, Leo Gagnon, Guillaume Lajoie

AI总结:

本文通过对比分布细化与基于任务奖励的学习,揭示了任务奖励在提升模型性能中的关键作用,实验表明任务奖励能带来更稳定的性能提升。

摘要:

前沿模型在整合基于任务奖励的强化学习(RL)到训练流程后,展现了卓越的能力,使系统从纯推理模型发展为复杂的智能体。然而,关于RL是否真正赋予基础模型新技能还是仅通过细化分布来激发潜在能力仍存在争议。为解决这一二元对立,本文通过RL作为工具,明确比较分布细化与基于任务奖励的学习。分析揭示了分布细化的固有局限,从原理上证明了最优解可能不利且方法本质上不稳定。此外,使用Llama-3.2-3B-Instruct、Qwen2.5-3B-Instruct和Qwen3-4B-Instruct-2507在数学数据集上的实验表明,细化带来的收益有限,而引入基于任务的奖励信号能显著帮助实现稳健的性能提升和稳定的训练。

英文摘要:

Frontier models have demonstrated exceptional capabilities following the integration of task-reward-based reinforcement learning (RL) into their training pipelines, enabling systems to evolve from pure reasoning models into sophisticated agents. However, debate persists regarding whether RL genuinely instills new skills within a base model or merely sharpens its existing distribution to elicit latent capabilities. To address this dichotomy, we present an explicit comparison between distribution sharpening and task-reward-based learning, utilizing RL as a tool to implement both paradigms. Our analysis reveals the inherent limitations of distribution sharpening, demonstrating from first principles how and why the optima can be unfavorable and the approach fundamentally unstable. Furthermore, our experiments using Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct and Qwen3-4B-Instruct-2507 on math datasets confirm that sharpening yields limited gains, whereas incorporating task-based reward signal can greatly help achieve robust performance improvements and stable learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑