利用Actor-Critic算法与ReLU网络合成程序化策略
Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks
AI总结：
本文提出利用ReLU网络与斜决策树的联系，将actor-critic算法学到的策略直接转换为程序化策略，无需PIRL专用算法，并在控制任务上取得优于现有PIRL算法的效果。
摘要：
程序可解释强化学习（PIRL）将策略编码为人类可读的计算机程序。最近提出的新算法旨在解决在程序化策略空间中缺乏梯度信号来引导搜索的问题。大多数此类PIRL算法首先训练一个神经策略，将其用作预言机（oracle）来引导程序化空间中的搜索。在本文中，我们表明，根据用于编码程序化策略的语言，并不需要这种PIRL特有的算法。这是因为我们可以使用actor-critic算法直接获得程序化策略。我们利用ReLU神经网络与斜决策树之间的联系，将通过actor-critic算法学到的策略转换为程序化策略。这种从ReLU网络的转换使我们能够合成编码在具有if-then-else结构、输入值的线性变换以及PID操作的程序中的策略。在多个控制问题上的实验结果表明，这种转换方法能够学习到简短且有效的策略。此外，转换后的策略至少具有竞争力，并且通常远优于PIRL算法合成的策略。
英文摘要：
Programmatically Interpretable Reinforcement Learning (PIRL) encodes policies in human-readable computer programs. Novel algorithms were recently introduced with the goal of handling the lack of gradient signal to guide the search in the space of programmatic policies. Most of such PIRL algorithms first train a neural policy that is used as an oracle to guide the search in the programmatic space. In this paper, we show that such PIRL-specific algorithms are not needed, depending on the language used to encode the programmatic policies. This is because one can use actor-critic algorithms to directly obtain a programmatic policy. We use a connection between ReLU neural networks and oblique decision trees to translate the policy learned with actor-critic algorithms into programmatic policies. This translation from ReLU networks allows us to synthesize policies encoded in programs with if-then-else structures, linear transformations of the input values, and PID operations. Empirical results on several control problems show that this translation approach is capable of learning short and effective policies. Moreover, the translated policies are at least competitive and often far superior to the policies PIRL algorithms synthesize.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
