arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.15845cs.LGcs.AI
收藏

拓扑经验回放

Topological Experience Replay

Massachusetts Institute of Technology(麻省理工学院);Aalto University(阿尔托大学)

Zhang-Wei Hong, Tao Chen, Yen-Chen Lin, Joni Pajarinen, Pulkit Agrawal

AI总结:

本文提出拓扑经验回放方法,通过将经验组织成图并利用广度优先搜索执行价值备份,解决了传统采样忽略后继状态Q值精度的问题,在多种目标达成任务上显著提升了数据效率。

摘要:

最先进的深度Q学习方法使用从经验回放缓冲区采样的状态转移元组来更新Q值。这种策略通常基于时间差分(TD)误差等度量进行均匀随机采样或优先采样。由于一个状态的Q值依赖于后继状态的Q值,此类采样策略在学习Q函数时可能效率低下。如果数据采样策略忽略下一个状态Q值估计的精度,可能会导致对Q值进行无用且通常不正确的更新。为了缓解这一问题,我们将智能体的经验组织成一个图,该图明确跟踪状态Q值之间的依赖关系。图中的每条边表示通过执行单个动作在两个状态之间发生的转移。我们通过广度优先搜索执行价值备份,该搜索从终端状态集合开始展开图中的顶点,并连续向后移动。我们在各种目标达成任务上实证表明,我们的方法在数据效率上显著优于几种基线方法。值得注意的是,所提出的方法也优于消耗更多批次训练经验的基线方法,并且能够处理来自图像等高维观测数据的输入。

英文摘要:

State-of-the-art deep Q-learning methods update Q-values using state transition tuples sampled from the experience replay buffer. This strategy often uniformly and randomly samples or prioritizes data sampling based on measures such as the temporal difference (TD) error. Such sampling strategies can be inefficient at learning Q-function because a state's Q-value depends on the Q-value of successor states. If the data sampling strategy ignores the precision of the Q-value estimate of the next state, it can lead to useless and often incorrect updates to the Q-values. To mitigate this issue, we organize the agent's experience into a graph that explicitly tracks the dependency between Q-values of states. Each edge in the graph represents a transition between two states by executing a single action. We perform value backups via a breadth-first search starting from that expands vertices in the graph starting from the set of terminal states and successively moving backward. We empirically show that our method is substantially more data-efficient than several baselines on a diverse range of goal-reaching tasks. Notably, the proposed method also outperforms baselines that consume more batches of training experience and operates from high-dimensional observational data such as images.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑