基于从粗到细强化学习的连续控制
Continuous Control with Coarse-to-fine Reinforcement Learning
AI总结：
针对强化学习算法难部署于真实环境的问题，提出从粗到细强化学习框架CRL及具体算法CQN，通过迭代离散动作空间选高Q值区间实现连续控制，在RLBench任务和真实操作中表现优异。
摘要：
尽管近期在提升强化学习（RL）算法的样本效率方面取得了进展，但设计一种可实际部署于真实世界环境的RL算法仍是一项挑战。本文提出了从粗到细强化学习（Coarse-to-fine Reinforcement Learning，CRL）框架，该框架训练RL智能体以从粗到细的方式聚焦连续动作空间，使稳定、样本高效的基于值的RL算法可用于细粒度连续控制任务。我们的核心思路是训练智能体通过迭代以下流程输出动作：（i）将连续动作空间离散化为多个区间；（ii）选择Q值最高的区间，在下一层级进一步离散化。随后我们在CRL框架内提出了一种具体的基于值的算法，名为从粗到细Q网络（Coarse-to-fine Q-Network，CQN）。我们的实验表明，在20项稀疏奖励的RLBench操作任务上，仅需中等数量的环境交互和专家演示，CQN的性能就显著优于RL和行为克隆基线。我们还证明，CQN仅需数分钟的在线训练，就能稳健地学会完成真实世界操作任务。
英文摘要：
Despite recent advances in improving the sample-efficiency of reinforcement learning (RL) algorithms, designing an RL algorithm that can be practically deployed in real-world environments remains a challenge. In this paper, we present Coarse-to-fine Reinforcement Learning (CRL), a framework that trains RL agents to zoom-into a continuous action space in a coarse-to-fine manner, enabling the use of stable, sample-efficient value-based RL algorithms for fine-grained continuous control tasks. Our key idea is to train agents that output actions by iterating the procedure of (i) discretizing the continuous action space into multiple intervals and (ii) selecting the interval with the highest Q-value to further discretize at the next level. We then introduce a concrete, value-based algorithm within the CRL framework called Coarse-to-fine Q-Network (CQN). Our experiments demonstrate that CQN significantly outperforms RL and behavior cloning baselines on 20 sparsely-rewarded RLBench manipulation tasks with a modest number of environment interactions and expert demonstrations. We also show that CQN robustly learns to solve real-world manipulation tasks within a few minutes of online training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
