基于好奇探索的目标条件离线规划
Goal-conditioned Offline Planning from Curious Exploration
AI总结：
本文针对无额外环境交互的离线场景，提出将基于模型的规划与基于图的价值聚合相结合，以纠正目标条件价值函数中的估计伪影，从而显著提升零样本目标到达性能。
摘要：
好奇心已成为深度强化学习中一种强大的探索策略。值得注意的是，利用预期的未来新颖性作为内在动机已被证明能够高效地生成探索轨迹，同时也能学习到鲁棒的动力学模型。我们考虑在没有任何额外环境交互的情况下，从此类无监督探索技术的产物中提取目标条件行为的挑战。我们发现，用于提取价值函数和策略的传统目标条件强化学习方法在这一困难的离线设置中表现不佳。通过分析最优目标条件价值函数的几何结构，我们将这一问题与学习价值中一类特定的估计伪影联系起来。为了减少这些伪影的出现，我们提出将基于模型的对学习价值景观的规划与基于图的价值聚合方案相结合。我们展示了这种组合如何能够同时纠正局部和全局伪影，在多种模拟环境中显著提升了零样本目标到达性能。
英文摘要：
Curiosity has established itself as a powerful exploration strategy in deep reinforcement learning. Notably, leveraging expected future novelty as intrinsic motivation has been shown to efficiently generate exploratory trajectories, as well as a robust dynamics model. We consider the challenge of extracting goal-conditioned behavior from the products of such unsupervised exploration techniques, without any additional environment interaction. We find that conventional goal-conditioned reinforcement learning approaches for extracting a value function and policy fall short in this difficult offline setting. By analyzing the geometry of optimal goal-conditioned value functions, we relate this issue to a specific class of estimation artifacts in learned values. In order to mitigate their occurrence, we propose to combine model-based planning over learned value landscapes with a graph-based value aggregation scheme. We show how this combination can correct both local and global artifacts, obtaining significant improvements in zero-shot goal-reaching performance across diverse simulated environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
