arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.21940cs.LGcs.AI
收藏

使用评论家引导进化的、以人类可读程序作为强化学习智能体执行器的方法

Human-Readable Programs as Actors of Reinforcement Learning Agents Using Critic-Moderated Evolution

Vrije Universiteit Brussel(布鲁塞尔自由大学)

Senne Deproost, Denis Steckelmacher, Ann Nowé

AI总结:

针对强化学习黑盒策略可解释性差、蒸馏程序性能低的问题,提出以TD3评论家为遗传算法目标函数、训练中直接合成程序策略的方法,兼具可解释性与高样本效率。

摘要:

随着深度强化学习(DRL)越来越多地被用于现实世界系统的控制,强化学习核心的神经网络缺乏透明度的问题引发了关注。程序式强化学习(PRL)能够以源代码的形式创建这种黑盒的表示,不仅提升了控制器的可解释性,还允许用户进行调整。但这类方法聚焦于将黑盒策略蒸馏为程序,且是在学习完成后基于生成行为与预期行为的均方误差完成蒸馏,舍弃了强化学习算法的其他要素,因此蒸馏后的策略性能可能远低于学习得到的黑盒策略。\n 本文提出直接学习一个程序作为强化学习智能体的策略。我们以TD3(双延迟深度确定性策略梯度)为基础,利用其评论家作为合成程序的遗传算法的目标函数基础。我们的方法在训练过程中构建程序,而非事后构建,这能引导程序追求实际高奖励,而非简单的均方误差。此外,我们的方法借助TD3评论家实现了高样本效率,不同于依赖蒙特卡洛评估的纯遗传方法。我们的实验在简单的网格世界环境中验证了该方法的有效性、可解释性和样本效率。

英文摘要:

With Deep Reinforcement Learning (DRL) being increasingly considered for the control of real-world systems, the lack of transparency of the neural network at the core of RL becomes a concern. Programmatic Reinforcement Learning (PRL) is able to to create representations of this black-box in the form of source code, not only increasing the explainability of the controller but also allowing for user adaptations. However, these methods focus on distilling a black-box policy into a program and do so after learning using the Mean Squared Error between produced and wanted behaviour, discarding other elements of the RL algorithm. The distilled policy may therefore perform significantly worse than the black-box learned policy. In this paper, we propose to directly learn a program as the policy of an RL agent. We build on TD3 and use its critics as the basis of the objective function of a genetic algorithm that syntheses the program. Our approach builds the program during training, as opposed to after the fact. This steers the program to actual high rewards, instead of a simple Mean Squared Error. Also, our approach leverages the TD3 critics to achieve high sample-efficiency, as opposed to pure genetic methods that rely on Monte-Carlo evaluations. Our experiments demonstrate the validity, explainability and sample-efficiency of our approach in a simple gridworld environment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑