DiPRL: 通过架构熵正则化学习离散程序性策略
DiPRL: Learning Discrete Programmatic Policies via Architecture Entropy Regularization
AI总结：
本文提出DiPRL，一种通过架构熵正则化学习可解释程序性策略的方法，以避免事后细化阶段，提高策略表达性和任务性能。
摘要：
程序性强化学习（PRL）通过将策略表示为可读可编辑的程序，为深度强化学习提供了一种可解释的替代方案。尽管基于梯度的方法已被开发用于优化程序的连续松弛，但在将连续松弛转换回离散程序时会显著降低性能。事后离散化会丢弃优化的分支和参数，导致策略表达性崩溃和任务性能下降，从而需要额外的微调。为克服这些限制，我们提出了可微离散程序性强化学习（DiPRL），一种在训练过程中使程序接近离散的方法，避免了单独的事后微调阶段。我们首先分析了基于梯度方法事后离散化引入的性能下降固有风险。然后，我们引入了程序架构熵正则化，这使得训练过程平滑且可微，鼓励收敛到离散程序。DiPRL在保持基于梯度优化效率的同时，减轻了事后离散化的风险。在多个离散和连续RL任务中的实验表明，DiPRL可以通过可解释的程序性策略实现强大的性能。
英文摘要：
Programmatic reinforcement learning (PRL) offers an interpretable alternative to deep reinforcement learning by representing policies as human-readable and -editable programs. While gradient-based methods have been developed to optimize continuous relaxations of programs, they face a significant performance drop when converting the continuous relaxations back into discrete programs. Post-hoc discretization can discard optimized branches and parameters in a program, which results in a collapse of policy expressivity and lowered task performance, leading in turn to a need for additional fine-tuning. To overcome these limitations, we propose Differentiable Discrete Programmatic Reinforcement Learning (DiPRL), a method that learns programmatic policies that become nearly discrete during training, avoiding a separate post-hoc fine-tuning stage. We first analyze the inherent risks of performance drop introduced by post-hoc discretization of gradient-based methods. Then, we introduce programmatic architecture entropy regularization, which enables smooth, differentiable training that encourages convergence toward a discrete program. DiPRL maintains the efficiency of gradient-based optimization while mitigating the risks of post-hoc discretization. Our experiments across multiple discrete and continuous RL tasks demonstrate that DiPRL can achieve strong performance via interpretable programmatic policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
