从过去到未来：重新思考资格迹
From Past to Future: Rethinking Eligibility Traces
AI总结：
本文提出双向价值函数，重新审视资格迹，以加速策略评估，并在某些挑战性场景下优于TD(λ)方法。
摘要：
在本文中，我们针对信用分配和策略评估的挑战提出了一个全新的视角。首先，我们深入探讨了资格迹的细微之处，并研究了其更新可能对先前状态产生意外信用分配的情况。通过这一研究，我们提出了一种新型价值函数的概念，我们称之为“双向价值函数”。与传统的状态价值函数不同，双向价值函数同时考虑未来期望回报（从当前状态起预期的奖励）和过去期望回报（从回合开始到当前的累积奖励）。我们推导出了学习该价值函数的原理性更新方程，并通过实验证明了其在增强策略评估过程中的有效性。特别是，我们的结果表明，所提出的学习方法在某些具有挑战性的情境下，可以比TD(λ)——一种直接学习前向价值函数v^π的方法——更快地进行策略评估。总体而言，我们的发现为资格迹及其所启发的新型价值函数带来了新的视角，并展示了其在策略评估方面的潜在优势。
英文摘要：
In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the \emph{bidirectional value function}. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD($λ$) -- a method that learns forward value functions, $v^π$, \emph{directly}. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
