超越模仿：从演示中恢复密集奖励
Beyond Imitation: Recovering Dense Rewards from Demonstrations
AI总结：
本研究证明监督微调（SFT）与逆强化学习等价，可从SFT模型中恢复token级密集奖励，提出的Dense-Path REINFORCE方法能利用该奖励优化策略，在指令跟随基准上表现优于原始SFT模型，重新定义了SFT的价值。
摘要：
传统上，监督微调（SFT）被视为一种简单的模仿学习过程，仅用于训练策略在演示数据集上模仿专家行为。在本研究中，我们通过建立SFT与逆强化学习之间的基本等价关系，对这一观点提出了挑战。我们证明了SFT目标是逆Q学习的一个特例，这意味着SFT过程不仅学习策略，还会学习一个隐含的、密集的、token级的奖励模型，该模型能够解释专家演示。随后，我们通过构建一个基于基线的奖励函数，展示了如何直接从SFT模型中恢复这种密集奖励信号。这类密集奖励模型的存在带来诸多益处，可为生成的每个token提供细粒度的信用分配。我们展示了一项关键应用：利用这些恢复的奖励，通过强化学习进一步优化策略。我们的方法Dense-Path REINFORCE在指令跟随基准上的表现始终优于原始SFT模型。本研究将SFT重新定义为一种强大的奖励学习机制，而非仅仅是策略模仿，为利用专家演示开辟了新的可能。
英文摘要：
Conventionally, supervised fine-tuning (SFT) is treated as a simple imitation learning process that only trains a policy to imitate expert behavior on demonstration datasets. In this work, we challenge this view by establishing a fundamental equivalence between SFT and Inverse Reinforcement Learning. We prove that the SFT objective is a special case of Inverse Q-Learning, which implies that the SFT process does not just learn a policy, but also an implicit, dense, token-level reward model that explains the expert demonstrations. We then show how to recover this dense reward signal directly from the SFT model by formulating a baseline-relative reward function. The availability of such a dense reward model offers numerous benefits, providing granular credit assignment for each token generated. We demonstrate one key application by using these recovered rewards to further improve the policy with reinforcement learning. Our method, Dense-Path REINFORCE, consistently outperforms the original SFT models on instruction-following benchmarks. This work reframes SFT not merely as policy imitation but as a powerful reward learning mechanism, opening new possibilities for leveraging expert demonstrations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
