使用地标映射状态空间以实现普遍目标
Mapping State Space using Landmarks for Universal Goal Reaching
摘要:
一个很好地理解环境的智能体应该能够将其技能应用于任何给定的目标,这就导致了学习通用价值函数近似器(UVFA)的基本问题。UVFA学习预测所有状态-目标对之间的累积奖励。然而,从经验上看,长期目标的价值函数总是很难估计,因此可能导致政策失败。这对神经网络的学习过程和能力提出了挑战。我们提出了一种方法来解决这个问题,在大型MDPs稀疏奖励,其中探索和路由跨越远程状态都是极具挑战性的。我们的方法显式地以分层的方式对环境进行建模,高级动态基于地标的地图抽象了访问的状态空间,而低级价值网络则用于获得精确的本地决策。我们使用最远点采样从过去的经验中选择地标状态,这与简单的均匀采样相比,提高了探索。实验表明,我们的方法使智能体能够在早期训练阶段达到长期目标,并在许多具有挑战性的任务中实现比标准RL算法更好的性能。
英文摘要:
An agent that has well understood the environment should be able to apply its skills for any given goals, leading to the fundamental problem of learning the Universal Value Function Approximator (UVFA). A UVFA learns to predict the cumulative rewards between all state-goal pairs. However, empirically, the value function for long-range goals is always hard to estimate and may consequently result in failed policy. This has presented challenges to the learning process and the capability of neural networks. We propose a method to address this issue in large MDPs with sparse rewards, in which exploration and routing across remote states are both extremely challenging. Our method explicitly models the environment in a hierarchical manner, with a high-level dynamic landmark-based map abstracting the visited state space, and a low-level value network to derive precise local decisions. We use farthest point sampling to select landmark states from past experience, which has improved exploration compared with simple uniform sampling. Experimentally we showed that our method enables the agent to reach long-range goals at the early training stage, and achieve better performance than standard RL algorithms for a number of challenging tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。