通过学习组合程序实现分层程序化强化学习
Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs
AI总结：
本文提出分层程序化强化学习框架HPRL，通过学习元策略组合程序嵌入空间中的程序，克服LEAPS受数据集分布限制和信用分配不足的问题，在Karel领域取得优于基线的效果。
摘要：
为了生成人类可解释且能更好地泛化到新场景的强化学习（RL）策略，Trivedi等人（2021）提出了一种方法（LEAPS），该方法首先学习一个程序嵌入空间，以连续参数化来自预生成程序数据集的多样化程序，然后在给定任务时在该学习到的程序嵌入空间中搜索求解任务的程序。尽管结果令人鼓舞，但LEAPS能够产生的程序策略受限于程序数据集的分布。此外，在搜索过程中，LEAPS仅根据回报来评估每个候选程序，无法精确奖励程序的正确部分并惩罚错误部分。为了解决这些问题，我们提出学习一个元策略，该策略组合一系列从学习到的程序嵌入空间中采样的程序。通过学习组合程序，我们提出的分层程序化强化学习（HPRL）框架能够产生描述分布外复杂行为的程序策略，并直接将信用分配给引发期望行为的程序。在Karel领域中的实验结果表明，我们提出的框架优于基线方法。消融研究证实了LEAPS的局限性，并证明了我们设计选择的合理性。
英文摘要：
Aiming to produce reinforcement learning (RL) policies that are human-interpretable and can generalize better to novel scenarios, Trivedi et al. (2021) present a method (LEAPS) that first learns a program embedding space to continuously parameterize diverse programs from a pre-generated program dataset, and then searches for a task-solving program in the learned program embedding space when given a task. Despite the encouraging results, the program policies that LEAPS can produce are limited by the distribution of the program dataset. Furthermore, during searching, LEAPS evaluates each candidate program solely based on its return, failing to precisely reward correct parts of programs and penalize incorrect parts. To address these issues, we propose to learn a meta-policy that composes a series of programs sampled from the learned program embedding space. By learning to compose programs, our proposed hierarchical programmatic reinforcement learning (HPRL) framework can produce program policies that describe out-of-distributionally complex behaviors and directly assign credits to programs that induce desired behaviors. The experimental results in the Karel domain show that our proposed framework outperforms baselines. The ablation studies confirm the limitations of LEAPS and justify our design choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
