Skill-Critic：为分层强化学习精炼已习得的技能
Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning
AI总结：
针对分层强化学习中离线数据习得的低层策略因覆盖率低、分布偏移不可靠的问题，提出Skill-Critic算法，结合高层技能选择微调低层策略，用离线演示隐空间初始化正则化双策略并行优化，在多稀疏奖励环境验证有效。
摘要：
分层强化学习（RL）可通过将策略在时间维度上抽象为多个层级，加速长时序决策过程。基于技能（即原始动作序列）的方法已在稀疏奖励环境中展现出可观效果，这类方法通常从离线数据中发现技能隐空间与策略。然而，由于演示数据覆盖率低或分布偏移，最终得到的低层策略可能并不可靠。作为解决方案，我们提出Skill-Critic算法，结合高层技能选择对低层策略进行微调。我们的Skill-Critic算法同时优化低层与高层策略；这些策略以从离线演示中学到的隐空间进行初始化与正则化，以引导并行策略优化。我们在多个稀疏奖励RL环境中验证了Skill-Critic，包括Gran Turismo Sport中的一项全新稀疏奖励自主赛车任务。实验表明，Skill-Critic的低层策略微调与演示引导正则化对实现良好性能至关重要。代码与视频可访问我们的网站获取：https://sites.google.com/view/skill-critic。
英文摘要：
Hierarchical reinforcement learning (RL) can accelerate long-horizon decision-making by temporally abstracting a policy into multiple levels. Promising results in sparse reward environments have been seen with skills, i.e. sequences of primitive actions. Typically, a skill latent space and policy are discovered from offline data. However, the resulting low-level policy can be unreliable due to low-coverage demonstrations or distribution shifts. As a solution, we propose the Skill-Critic algorithm to fine-tune the low-level policy in conjunction with high-level skill selection. Our Skill-Critic algorithm optimizes both the low-level and high-level policies; these policies are initialized and regularized by the latent space learned from offline demonstrations to guide the parallel policy optimization. We validate Skill-Critic in multiple sparse-reward RL environments, including a new sparse-reward autonomous racing task in Gran Turismo Sport. The experiments show that Skill-Critic's low-level policy fine-tuning and demonstration-guided regularization are essential for good performance. Code and videos are available at our website: https://sites.google.com/view/skill-critic.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
