关于策略梯度推导中
On the "Causality" Step in Policy Gradient Derivations: A Pedagogical Reconciliation of Full Return and Reward-to-Go
AI总结:
本文通过前缀轨迹分布和分数函数恒等式明确推导了奖励到时的数学依据,澄清了完整回报与奖励到时之间的关系,使因果性论证成为推导的自然结果。
摘要:
在策略梯度的入门介绍中,通常通过完整轨迹回报推导REINFORCE估计器,然后通过
英文摘要:
In introductory presentations of policy gradients, one often derives the REINFORCE estimator using the full trajectory return and then states, by ``causality,'' that the full return may be replaced by the reward-to-go. Although this statement is correct, it is frequently presented at a level of rigor that leaves unclear where the past-reward terms disappear. This short paper isolates that step and gives a mathematically explicit derivation based on prefix trajectory distributions and the score-function identity. The resulting account does not change the estimator. Its contribution is conceptual: instead of presenting reward-to-go as a post hoc unbiased replacement for full return, it shows that reward-to-go arises directly once the objective is decomposed over prefix trajectories. In this formulation, the usual causality argument is recovered as a corollary of the derivation rather than as an additional heuristic principle.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。