We introduce a sampling-based learning method for solving optimal control problems involving task satisfaction constraints for systems with partially known dynamics. The control problems are defined by a cost to be minimized and a task to be satisfied, given in the language of signal temporal logic (STL). The complex nature of possible tasks generally makes them difficult to satisfy through random exploration, which limits the practical feasibility of the learning algorithm. Recent work has shown, however, that using a controller to guide the learning process by leveraging available knowledge of system dynamics to aid task satisfaction is greatly beneficial for improving the sample efficiency of the method. Motivated by these findings, this work introduces a controller derivation framework which naturally leads to computationally efficient controllers capable of offering such guidance during the learning process. The derived controllers aim to satisfy a set of so-called robustness specifications or funnels that are imposed on the temporal evolutions of the atomic propositions composing the STL task. Ideally, these specifications are prescribed in a way such that their satisfaction would lead to satisfaction of the STL task. In practice, however, such ideal funnels are not necessarily known a priori, and the guidance the controller offers depends on their estimates. This issue is hereby addressed by introducing an adaptation scheme for automatically updating the funnels during the learning procedure, thus diminishing the role of their initial, user-specified values. The effectiveness of the resulting learning algorithm is demonstrated by two simulation case studies.
This paper studies the online control synthesis problem for uncertain discrete-time systems subject to signal temporal logic (STL) specifications. Different from existing techniques, this work proposes an approach based on STL, reachability analysis, and temporal logic trees. Firstly, a real-time version of STL semantics and a tube-based temporal logic tree (tTLT) are proposed. We show that the tTLT is an underapproximation for the STL formula, in the sense that a trajectory satisfying an tTLT also satisfies the corresponding STL formula. Secondly, an online control synthesis algorithm is desi…
We present a robust control framework for time-critical systems in which satisfying real-time constraints robustly is of utmost importance for the safety of the system. Signal Temporal Logic (STL) provides a formal means to express a large variety of real-time constraints over signals and is suited for planning and control purposes as it allows us to reason about the time robustness of such constraints. The time robustness of STL particularly quantifies the extent to which timing uncertainties can be tolerated without violating real-time specifications. In this paper, we first pose a control p…
Feasibility-aware Learning of Robust Temporal Logic Controllers using BarrierNet
Amazon Robotics（亚马逊机器人）；Department of Mechanical Engineering（机械工程系）；Worcester Polytechnic Institute（沃斯堡技术学院）；Department of Electrical and Computer Engineering and the Department of Computer Science（电气与计算机工程系和计算机科学系）
We apply deep reinforcement learning (DRL) to design of a networked controller with network delays to complete a temporal control task that is described by a signal temporal logic (STL) formula. STL is useful to deal with a specification with a bounded time interval for a dynamical system. In general, an agent needs not only the current system state but also the past behavior of the system to determine a desired control action for satisfying the given STL formula. Additionally, we need to consider the effect of network delays for data transmissions. Thus, we propose an extended Markov decision…
We propose a framework based on Recurrent Neural Networks (RNNs) to determine an optimal control strategy for a discrete-time system that is required to satisfy specifications given as Signal Temporal Logic (STL) formulae. RNNs can store information of a system over time, thus, enable us to determine satisfaction of the dynamic temporal requirements specified in STL formulae. Given a STL formula, a dataset of satisfying system executions and corresponding control policies, we can use RNNs to predict a control policy at each time based on the current and previous states of system. We use Contro…
Prescribed Performance Control Guided Policy Improvement for Satisfying Signal Temporal Logic Tasks
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（决策与控制系统系，电气工程学院和计算机科学学院，皇家理工学院）
There are spatio-temporal rules that dictate how robots should operate in complex environments, e.g., road rules govern how (self-driving) vehicles should behave on the road. However, seamlessly incorporating such rules into a robot control policy remains challenging especially for real-time applications. In this work, given a desired spatio-temporal specification expressed in the Signal Temporal Logic (STL) language, we propose a semi-supervised controller synthesis technique that is attuned to human-like behaviors while satisfying desired STL specifications. Offline, we synthesize a trajecto…
We present a model-free reinforcement learning algorithm to find an optimal policy for a finite-horizon Markov decision process while guaranteeing a desired lower bound on the probability of satisfying a signal temporal logic (STL) specification. We propose a method to effectively augment the MDP state space to capture the required state history and express the STL objective as a reachability objective. The planning problem can then be formulated as a finite-horizon constrained Markov decision process (CMDP). For a general finite horizon CMDP problem with unknown transition probability, we dev…
