程序机器策略：通过集成程序合成与状态机解决长时程任务
Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machines
AI总结：
本文提出程序机器策略（POMP），融合程序化RL与状态机，通过检索多样程序作为模式并学习转移函数，解决长时程任务，优于现有基线且可泛化至更长时程。
摘要：
深度强化学习（深度RL）在多个领域表现出色，但缺乏泛化性和可解释性。另一方面，程序化RL方法（Trivedi等人，2021；Liu等人，2023）将RL任务重新表述为合成可在环境中执行的可解释程序。尽管取得了令人鼓舞的结果，这些方法仅限于短时程任务。另一方面，使用状态机（Inala等人，2020）表示RL策略可以归纳地泛化到长时程任务；然而，它难以扩展到获取多样且复杂的行为。这项工作提出了程序机器策略（POMP），它桥接了程序化RL和状态机策略的优势，允许表示复杂行为并解决长期任务。具体来说，我们引入了一种方法，可以检索一组有效、多样且兼容的程序。然后，我们使用这些程序作为状态机的模式，并学习一个转移函数在模式程序之间进行转移，从而能够捕获重复行为。我们提出的框架在各种任务上优于程序化RL和深度RL基线，并展示了无需任何微调即可归纳泛化到更长时程的能力。消融研究验证了我们提出的用于检索一组程序作为模式的搜索算法的有效性。
英文摘要：
Deep reinforcement learning (deep RL) excels in various domains but lacks generalizability and interpretability. On the other hand, programmatic RL methods (Trivedi et al., 2021; Liu et al., 2023) reformulate RL tasks as synthesizing interpretable programs that can be executed in the environments. Despite encouraging results, these methods are limited to short-horizon tasks. On the other hand, representing RL policies using state machines (Inala et al., 2020) can inductively generalize to long-horizon tasks; however, it struggles to scale up to acquire diverse and complex behaviors. This work proposes the Program Machine Policy (POMP), which bridges the advantages of programmatic RL and state machine policies, allowing for the representation of complex behaviors and the address of long-term tasks. Specifically, we introduce a method that can retrieve a set of effective, diverse, and compatible programs. Then, we use these programs as modes of a state machine and learn a transition function to transition among mode programs, allowing for capturing repetitive behaviors. Our proposed framework outperforms programmatic RL and deep RL baselines on various tasks and demonstrates the ability to inductively generalize to even longer horizons without any fine-tuning. Ablation studies justify the effectiveness of our proposed search algorithm for retrieving a set of programs as modes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
