非平稳环境下带策略撤退的非参数随机策略梯度
Non-Parametric Stochastic Policy Gradient with Strategic Retreat for Non-Stationary Environment
摘要：
在现代机器人技术中，在动态变化的环境下有效地计算最优控制策略对现成的参数策略梯度方法提出了实质性挑战，例如深度确定性策略梯度（DDPG）和双延迟深度确定性策略梯度（TD 3）。在本文中，我们提出了一个系统的方法来动态学习序列的最优控制政策的非参数，同时自主适应不断变化的环境动态。具体来说，我们的非参数基于内核的方法嵌入了一个政策分布的功能在一个非递减的欧几里得空间，因此允许其搜索空间被定义为一个非常高（可能无限）维RKHS（再生核希尔伯特空间）。此外，通过利用在RKHS中计算的相似性度量，我们用AdaptiveH技术增强了我们的非参数学习-自适应地选择一个时间帧窗口来完成在一些先前观察到的状态上采样的整个动作序列的最佳部分。为了验证我们提出的方法，我们进行了广泛的实验，多个经典的基准和一个模拟机器人基准配备了动态变化的环境。总体而言，我们的方法在学习表现方面远远优于成熟的DDPG和TD 3方法。
英文摘要：
In modern robotics, effectively computing optimal control policies under dynamically varying environments poses substantial challenges to the off-the-shelf parametric policy gradient methods, such as the Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic policy gradient (TD3). In this paper, we propose a systematic methodology to dynamically learn a sequence of optimal control policies non-parametrically, while autonomously adapting with the constantly changing environment dynamics. Specifically, our non-parametric kernel-based methodology embeds a policy distribution as the features in a non-decreasing Euclidean space, therefore allowing its search space to be defined as a very high (possible infinite) dimensional RKHS (Reproducing Kernel Hilbert Space). Moreover, by leveraging the similarity metric computed in RKHS, we augmented our non-parametric learning with the technique of AdaptiveH- adaptively selecting a time-frame window of finishing the optimal part of whole action-sequence sampled on some preceding observed state. To validate our proposed approach, we conducted extensive experiments with multiple classic benchmarks and one simulated robotics benchmark equipped with dynamically changing environments. Overall, our methodology has outperformed the well-established DDPG and TD3 methodology by a sizeable margin in terms of learning performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
