本文提出LLM-GS框架，通过Pythonic-DSL策略和Scheduled Hill Climbing算法，利用大语言模型引导搜索高效合成程序化强化学习策略，显著提升样本效率并降低使用门槛。
摘要：
程序化强化学习（PRL）已被探索用于通过程序来表示策略，以此实现可解释性和泛化能力。尽管已有成果显示出良好前景，但当前最先进的PRL方法仍受样本效率低下的制约，通常需要数千万次程序与环境的交互。为应对这一挑战，我们提出了一种新颖的LLM引导搜索框架（LLM-GS）。我们的核心思路是利用大语言模型（LLM）的编程专长和常识推理能力，来提升无假设、随机猜测式搜索方法的效率。针对LLM难以在领域特定语言（DSL）中生成精确且语法正确程序的问题，我们提出了一种Pythonic-DSL策略——先让LLM生成Python代码，再将其转换为DSL程序。为进一步优化LLM生成的程序，我们开发了一种名为Scheduled Hill Climbing的搜索算法，旨在高效探索程序化搜索空间，从而持续改进程序。在Karel领域的实验结果表明，我们的LLM-GS框架具有优越的有效性和效率。大量消融研究进一步验证了Pythonic-DSL策略和Scheduled Hill Climbing算法的关键作用。此外，我们在两个新任务上进行了实验，结果表明LLM-GS能够使不具备编程技能以及不了解领域或DSL知识的用户，通过自然语言描述任务即可获得高性能程序。
英文摘要：
Programmatic reinforcement learning (PRL) has been explored for representing policies through programs as a means to achieve interpretability and generalization. Despite promising outcomes, current state-of-the-art PRL methods are hindered by sample inefficiency, necessitating tens of millions of program-environment interactions. To tackle this challenge, we introduce a novel LLM-guided search framework (LLM-GS). Our key insight is to leverage the programming expertise and common sense reasoning of LLMs to enhance the efficiency of assumption-free, random-guessing search methods. We address the challenge of LLMs' inability to generate precise and grammatically correct programs in domain-specific languages (DSLs) by proposing a Pythonic-DSL strategy - an LLM is instructed to initially generate Python codes and then convert them into DSL programs. To further optimize the LLM-generated programs, we develop a search algorithm named Scheduled Hill Climbing, designed to efficiently explore the programmatic search space to improve the programs consistently. Experimental results in the Karel domain demonstrate our LLM-GS framework's superior effectiveness and efficiency. Extensive ablation studies further verify the critical role of our Pythonic-DSL strategy and Scheduled Hill Climbing algorithm. Moreover, we conduct experiments with two novel tasks, showing that LLM-GS enables users without programming skills and knowledge of the domain or DSL to describe the tasks in natural language to obtain performant programs.
We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents policies using a high-level, domain-specific programming language. Such programmatic policies have the benefits of being more easily interpreted than neural networks, and being amenable to verification by symbolic methods. We propose a new method, called Neurally Directed Program Sear…
We study the problem of programmatic reinforcement learning, in which policies are represented as short programs in a symbolic language. Programmatic policies can be more interpretable, generalizable, and amenable to formal verification than neural policies; however, designing rigorous learning approaches for such policies remains a challenge. Our approach to this challenge -- a meta-algorithm called PROPEL -- is based on three insights. First, we view our learning task as optimization in policy space, modulo the constraint that the desired policy has a programmatic representation, and solve t…
InnateCoder: Learning Programmatic Options with Foundation Models
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）；Eindhoven University of Technology（埃因霍温理工大学）；Centrum Wiskunde & Informatica（数学与计算机科学中心）
针对强化学习从零开始学习技能效率低的问题，提出InnateCoder系统，零样本利用基础模型的人类知识学习程序化选项并组合成复杂策略，在MicroRTS和Karel the Robot上验证了更高的样本效率。
Code Evolution for Control: Synthesizing Policies via LLM-Driven Evolutionary Search
Shenzhen Yuyi Tech. Co. Ltd.（深圳优衣科技有限公司）；City University of Hong Kong（香港城市大学）；Shenzhen University（深圳大学）；University of Electronic Science and Technology of China（电子科技大学）
The capability of a reinforcement learning (RL) agent heavily depends on the diversity of the learning scenarios generated by the environment. Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. The RTS environments are characterized by intelligent entities/non-RL agents cooperating and competing with the RL agents with large state and action spaces over a long period of time, resulting in an infinite space of feasible, but not necessarily realistic, scenarios involving complex interaction among different RL and non-RL agents. Yet, most of the ex…
Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；University of Trento（特伦托大学）
Being able to provide counterfactual interventions - sequences of actions we would have had to take for a desirable outcome to happen - is essential to explain how to change an unfavourable decision by a black-box machine learning model (e.g., being denied a loan request). Existing solutions have mainly focused on generating feasible interventions without providing explanations on their rationale. Moreover, they need to solve a separate optimization problem for each user. In this paper, we take a different approach and learn a program that outputs a sequence of explainable counterfactual actio…
University of Southern California（南加利福尼亚大学）；NAVER AI Lab（NAVER人工智能实验室）
Recently, deep reinforcement learning (DRL) methods have achieved impressive performance on tasks in a variety of domains. However, neural network policies produced with DRL methods are not human-interpretable and often have difficulty generalizing to novel scenarios. To address these issues, prior works explore learning programmatic policies that are more interpretable and structured for generalization. Yet, these works either employ limited policy representations (e.g. decision trees, state machines, or predefined program templates) or require stronger supervision (e.g. input/output state pa…
Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification constraints during learning. We present an iterative framework, MORL, for improving the learned policies using program synthesis. Concretely, we propose to use synthesis techniques to obtain a symbolic representation of the learned policy, which can then be debugged manually or automatically using program repair. After the repair step, we use behavior cloning to obtain the policy…
California Institute of Technology（加州理工学院）；Microsoft Research（微软研究院）
We study the problem of controllable generation of long-term sequential behaviors, where the goal is to calibrate to multiple behavior styles simultaneously. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are two questions that pose significant challenges when generating long-term behaviors: how should we specify the factors of variation to control, and how can we ensure that the generated behavior faithfully demonstrates combinatorially many styles? We leverage programmatic labeling functions to specify controllable styles, and derive a for…
This paper presents a reinforcement learning approach to synthesizing task-driven control policies for robotic systems equipped with rich sensory modalities (e.g., vision or depth). Standard reinforcement learning algorithms typically produce policies that tightly couple control actions to the entirety of the system's state and rich sensor observations. As a consequence, the resulting policies can often be sensitive to changes in task-irrelevant portions of the state or observations (e.g., changing background colors). In contrast, the approach we present here learns to create a task-driven rep…
Robots equipped with rich sensing modalities (e.g., RGB-D cameras) performing long-horizon tasks motivate the need for policies that are highly memory-efficient. State-of-the-art approaches for controlling robots often use memory representations that are excessively rich for the task or rely on hand-crafted tricks for memory efficiency. Instead, this work provides a general approach for jointly synthesizing memory representations and policies; the resulting policies actively seek to reduce memory requirements. Specifically, we present a reinforcement learning framework that leverages an implem…
Electronic Arts（美国艺电）；NVIDIA（英伟达）；Max-Planck-Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Toronto（多伦多大学）；Simon Fraser University（西蒙弗雷泽大学）
Movement is how people interact with and affect their environment. For realistic character animation, it is necessary to synthesize such interactions between virtual characters and their surroundings. Despite recent progress in character animation using machine learning, most systems focus on controlling an agent's movements in fairly simple and homogeneous environments, with limited interactions with other objects. Furthermore, many previous approaches that synthesize human-scene interactions require significant manual labeling of the training data. In contrast, we present a system that uses …
Columbia University（哥伦比亚大学）；Massachusetts Institute of Technology（麻省理工学院）
The problem of retrosynthetic planning can be framed as one player game, in which the chemist (or a computer program) works backwards from a molecular target to simpler starting materials though a series of choices regarding which reactions to perform. This game is challenging as the combinatorial space of possible choices is astronomical, and the value of each choice remains uncertain until the synthesis plan is completed and its cost evaluated. Here, we address this problem using deep reinforcement learning to identify policies that make (near) optimal reaction choices during each step of re…
The problem of synthesizing stochastic explicit model predictive control policies is known to be quickly intractable even for systems of modest complexity when using classical control-theoretic methods. To address this challenge, we present a scalable alternative called stochastic parametric differentiable predictive control (SP-DPC) for unsupervised learning of neural control policies governing stochastic linear systems subject to nonlinear chance constraints. SP-DPC is formulated as a deterministic approximation to the stochastic parametric constrained optimal control problem. This formulati…
Deep ensembles have been shown to extend the positive effect seen in typical ensemble learning to neural networks and to reinforcement learning (RL). However, there is still much to be done to improve the efficiency of such ensemble models. In this work, we present Diverse Ensembles for Fast Transfer in RL (DEFT), a new ensemble-based method for reinforcement learning in highly multimodal environments and improved transfer to unseen environments. The algorithm is broken down into two main phases: training of ensemble members, and synthesis (or fine-tuning) of the ensemble members into a policy…
1-2-3-Go! Policy Synthesis for Parameterized Markov Decision Processes via Decision-Tree Learning and Generalization
Technical University of Munich（慕尼黑工业大学）；Vali-e-Asr University of Rafsanjan（拉夫桑贾德瓦利阿斯尔大学）；Masaryk University（马萨里克大学）；Institute of Science and Technology（科学技术研究院）
University of Pennsylvania（宾夕法尼亚大学）；Deapartment of Computer and Information Science（计算机与信息科学系）
Reinforcement learning is a promising approach to synthesizing policies for challenging robotics tasks. A key problem is how to ensure safety of the learned policy---e.g., that a walking robot does not fall over or that an autonomous car does not run into an obstacle. We focus on the setting where the dynamics are known, and the goal is to ensure that a policy trained in simulation satisfies a given safety constraint. We propose an approach, called model predictive shielding (MPS), that switches on-the-fly between a learned policy and a backup policy to ensure safety. We prove that our approac…
Reinforcement learning with timed constraints for robotics motion planning
Department of Mechanical Engineering, Iowa Technology Institute, University of Iowa（机械工程系，Iowa技术研究所，爱荷华大学）；Talus Renewables, Inc.（Talus Renewables公司）；Department of Industrial, Electronic, and Mechanical Engineering, Roma Tre University（工业、电子和机械工程系，罗马三大学）
There has been significant recent interest in devising verification techniques for learning-enabled controllers (LECs) that manage safety-critical systems. Given the opacity and lack of interpretability of the neural policies that govern the behavior of such controllers, many existing approaches enforce safety properties through shield, a dynamic monitoring-and-repairing mechanism that ensures a LEC does not emit actions that would violate desired safety conditions. These methods, however, have been shown to have significant scalability limitations because verification costs grow as problem di…
