arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12874cs.LG
收藏

技能还是运气?基于优势函数的回报分解

Skill or Luck? Return Decomposition via Advantage Functions

Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)

Hsiao-Ru Pan, Bernhard Schölkopf

AI总结:

本文利用优势函数作为动作对回报的因果效应,将轨迹回报分解为技能与运气,并提出离策略DAE,在无需重要性采样或动作截断的情况下实现高效离策略学习。

摘要:

从离策略数据中学习对于样本高效的强化学习至关重要。在本工作中,我们基于如下洞见:优势函数可以理解为某个动作对回报的因果效应,并证明这使我们能够将一条轨迹的回报分解为由智能体动作所导致的部分(技能)和不在智能体控制范围内的部分(运气)。此外,这种分解使我们能够自然地将直接优势估计(DAE)扩展到离策略设置(离策略DAE)。所得到的方法能够从离策略轨迹中学习,而无需依赖重要性采样技术或截断离策略动作。我们将离策略DAE与先前方法建立联系,以展示它如何能够加速学习,以及所提出的离策略修正在何时是重要的。最后,我们使用MinAtar环境说明忽略离策略修正如何导致次优的策略优化性能。

英文摘要:

Learning from off-policy data is essential for sample-efficient reinforcement learning. In the present work, we build on the insight that the advantage function can be understood as the causal effect of an action on the return, and show that this allows us to decompose the return of a trajectory into parts caused by the agent's actions (skill) and parts outside of the agent's control (luck). Furthermore, this decomposition enables us to naturally extend Direct Advantage Estimation (DAE) to off-policy settings (Off-policy DAE). The resulting method can learn from off-policy trajectories without relying on importance sampling techniques or truncating off-policy actions. We draw connections between Off-policy DAE and previous methods to demonstrate how it can speed up learning and when the proposed off-policy corrections are important. Finally, we use the MinAtar environments to illustrate how ignoring off-policy corrections can lead to suboptimal policy optimization performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑