arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.04518cs.AIcs.LG
收藏

无需扩展的A*搜索:用深度Q网络学习启发式函数

A* Search Without Expansions: Learning Heuristic Functions with Deep Q-Networks

Department of Computer Science and Engineering(计算机科学与工程系);Faculty of Computer and Information Science(计算机与信息科学系);Department of Computer Science(计算机科学系)

Forest Agostinelli, Shahaf S. Shperberg, Alexander Shmakov, Stephen McAleer, Roy Fox, Pierre Baldi

AI总结:

针对大动作空间下A*搜索计算成本高的问题,提出基于Q值的Q*搜索算法,结合深度Q网络学习启发式函数,在保证最优解的同时大幅提升搜索速度、减少节点生成量。

摘要:

使用A*搜索高效求解大动作空间的问题仍是一项重大挑战。这是因为在A*搜索的每次迭代中,生成的节点数量和启发式函数的调用次数都会随动作空间的规模线性增长。当A*搜索使用由深度神经网络这类计算成本高昂的函数逼近器学习得到的启发式函数时,这一负担会更加明显。为解决该问题,我们提出Q*搜索算法,它利用的启发式函数接收一个状态后,仅需单次函数调用,就能返回该状态下所有可能转移的代价-去向估计,以及对应转移代价的估计——无需执行转移或生成后继状态;这类动作-状态估计通常被称为Q值。这大幅降低了计算时间和内存占用。此外,我们证明,当启发式函数不会高估转移代价与状态代价-去向之和时,Q*搜索保证能找到最短路径。为获取Q*搜索所需的启发式函数,我们采用深度Q网络架构,在无任何先验知识的情况下,通过与领域交互学习状态-动作启发式函数。我们在不同领域和动作空间下将Q*与学习到的启发式函数结合使用,结果表明随着动作空间规模增大,Q*仅产生很小的运行时开销。此外,实验结果显示,Q*搜索的速度最高可达A*搜索的129倍,生成的节点数最多比A*搜索少1288倍。

英文摘要:

Efficiently solving problems with large action spaces using A* search remains a significant challenge. This is because, for each iteration of A* search, the number of nodes generated and the number of heuristic function applications grow linearly with the size of the action space. This burden becomes even more apparent when A* search uses a heuristic function learned by computationally expensive function approximators, such as deep neural networks. To address this issue, we introduce Q*, a search algorithm that leverages heuristics capable of receiving a state and, in a single function call, returning cost-to-go estimates for all possible transitions from that state, along with estimates of the corresponding transition costs -- without the need to apply the transitions or generate the successor states; such action-state estimation are typically known as Q-values. This significantly reduces computation time and memory usage. In addition, we prove that Q* search is guaranteed to find a shortest path given a heuristic function that does not overestimate the sum of the transition cost and cost-to-go of the state. To obtain heuristics for Q* search, we employ a deep Q-network architecture to learn a state-action heuristic function from domain interaction, without any prior knowledge. We use Q* with our learned heuristic on different domains and action spaces, showing that Q* suffers from only a small runtime overhead as the size of the action space increases. In addition, our empirical results show Q* search is up to 129 times faster and generates up to 1288 times fewer nodes than A* search.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑