Model-free control based on the idea of Reinforcement Learning is a promising approach that has recently gained extensive attention. However, Reinforcement-Learning-based control methods solely focus on the regulation problem or learn to track a reference that is generated by a time-invariant exo-system. In the latter case, controllers are only able to track the time-invariant reference dynamics which they have been trained on and need to be re-trained each time the reference dynamics change. Consequently, these methods fail in a number of applications which obviously rely on a trajectory not being generated by an exo-system. One prominent example is autonomous driving. This paper provides for the first time an adaptive optimal control method capable to track reference trajectories not being generated by a time-invariant exo-system. The main innovation is a novel Q-function that directly incorporates a given reference trajectory on a moving horizon. This new Q-function exhibits a particular structure which allows the design of an efficient, iterative, provably convergent Reinforcement Learning algorithm that enables optimal tracking. Two real-world examples demonstrate the effectiveness of our new method.
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；IAV GmbH（IAV有限公司）
In recent years, reinforcement learning (RL) has gained increasing attention in control engineering. Especially, policy gradient methods are widely used. In this work, we improve the tracking performance of proximal policy optimization (PPO) for arbitrary reference signals by incorporating information about future reference values. Two variants of extending the argument of the actor and the critic taking future reference values into account are presented. In the first variant, global future reference values are added to the argument. For the second variant, a novel kind of residual space with …
This paper presents a novel model-reference reinforcement learning algorithm for the intelligent tracking control of uncertain autonomous surface vehicles with collision avoidance. The proposed control algorithm combines a conventional control method with reinforcement learning to enhance control accuracy and intelligence. In the proposed control design, a nominal system is considered for the design of a baseline tracking controller using a conventional control approach. The nominal system also defines the desired behaviour of uncertain autonomous surface vehicles in an obstacle-free environme…
Reinforcement learning (RL) is a promising, upcoming topic in automatic control applications. Where classical control approaches require a priori system knowledge, data-driven control approaches like RL allow a model-free controller design procedure, rendering them emergent techniques for systems with changing plant structures and varying parameters. While it was already shown in various applications that the transient control behavior for complex systems can be sufficiently handled by RL, the challenge of non-vanishing steady-state control errors remains, which arises from the usage of contro…
This paper presents a novel reference governor scheme for overshoot mitigation in tracking control systems. Our proposed scheme, referred to as the Reference Governor with Dynamic Constraint (RG-DC), recasts the overshoot mitigation problem as a constraint management problem. The outcome of this reformulation is a dynamic Maximal Admissible Set (MAS), which varies in real-time as a function of the reference signal and the tracking output. The RG-DC employs the dynamic MAS to modify the reference signal to mitigate or, if possible, prevent overshoot. We present several properties of the dynamic…
Most passivity based trajectory tracking algorithms for mechanical systems can only stabilise reference trajectories that have constant energy. This paper overcomes this limitation by deriving a single variable Hamiltonian model for the reference trajectory and solving along the constrained trajectory to obtain a reference potential. This potential is then used as the model to shape the energy of the true system such that its free solutions include the desired reference trajectory. The proposed trajectory tracking algorithm interconnects the reference and true systems through a virtual spring …
This paper studies the tracking control problem for nonholonomic mobile robots based on second order dynamics, with application to consensus tracking and formation tracking. The greatest novelty in this paper is that the reference trajectory can be arbitrarily chosen, in the sense that the condition of persistency excitation and any other requirements are not imposed on the leader. In this paper, at first, the tracking control of one leader with single follower is taken into consideration, which is converted to the stabilization of two relative subsystems by the design of an adjoint system. Th…
Various efforts have been devoted to developing stabilizing distributed Model Predictive Control (MPC) schemes for tracking piecewise constant references. In these schemes, terminal sets are usually computed offline and used in the MPC online phase to guarantee recursive feasibility and asymptotic stability. Maximal invariant terminal sets do not necessarily respect the distributed structure of the network, hindering the distributed implementation of the controller. On the other hand, ellipsoidal terminal sets respect the distributed structure, but may lead to conservative schemes. In this pap…
Distributed average tracking for multiple reference signals with general linear dynamics
State Key Laboratory for Turbulence and Complex Systems, Department of Mechanics and Engineering Science, College of Engineering, Peking University（北京大学工学院力学与工程科学系湍流国家重点实验室）
In precision aquaculture, the primary goal is to maximize biomass production while minimizing production costs. This objective can be achieved by optimizing factors that have a strong influence on fish growth, such as the feeding rate, temperature, and dissolved oxygen. This paper provides a comparative study of three model predictive control (MPC) strategies for fish growth reference tracking under a representative bioenergetic growth model in precision aquaculture. We propose to evaluate three candidate MPC formulations for fish growth reference tracking based on the receding horizon. The fi…
This letter presents a data-based control design for reference tracking applications. This design finds the optimal control sequence, which minimizes a quadratic cost function consisting of tracking error and input increments over a finite interval [0,N]. The only information needed is the first N+1 Markov parameters of the system. This design is employed on a tensegrity morphing airfoil whose topology has been described in detail in this letter. A NACA 2412 airfoil with specified morphing targets is chosen to verify the developed design. The principle developed in this letter is also applicab…
In this paper, an approach for active power control of individual wind turbines is presented. State-of-the-art controllers typically employ separate control loops for torque and pitch control. In contrast, we use a multivariable control approach. In detail, active power control is achieved by using reference trajectories for generator speed, generator torque, and pitch angle such that a desired power demand is met if weather conditions allow. Then, a linear quadratic (LQ) optimal controller is used for reference tracking. In an OpenFAST simulation environment, the controller is compared to a s…
This paper studies a leader-follower formation tracking problem where the leaders are moving at the same unknown bounded velocity. A distance-based control law is proposed for follower agents to maintain the desired distances in the formation and move at the leaders' velocity. The control law consists of a component to handle the uncertainty of the leaders' velocity and a component to achieve the desired distances in finite time. Numerical simulations are also provided to support the theoretical results.
University of Stuttgart（斯图加特大学）；ETH Zurich（苏黎世联邦理工学院）
In this paper, we present a method to analyze local and global stability in offset-free setpoint tracking using neural network controllers and we provide ellipsoidal inner approximations of the corresponding region of attraction. We consider a feedback interconnection of a linear plant in connection with a neural network controller and an integrator, which allows for offset-free tracking of a desired piecewise constant reference that enters the controller as an external input. Exploiting the fact that activation functions used in neural networks are slope-restricted, we derive linear matrix in…
In Model Predictive Control (MPC) formulations of trajectory tracking problems, infeasible reference trajectories and a-priori unknown constraints can lead to cumbersome designs, aggressive tracking, and loss of recursive feasibility. This is the case, for example, in trajectory tracking applications for mobile systems in the presence of constraints which are not fully known a-priori. In this paper, we propose a new framework called Model Predictive Flexible trajectory Tracking Control (MPFTC), which relaxes the trajectory tracking requirement. Additionally, we accommodate recursive feasibilit…
A distributed model predictive control scheme is developed for tracking piecewise constant references where the terminal set is reconfigured online, whereas the terminal controller is computed offline. Unlike many standard existing schemes, this scheme yields large feasible regions without performing offline centralized computations. Although the resulting optimal control problem (OCP) is a semidefinite program (SDP), an SDP scalability method based on diagonal dominance is used to approximate the derived SDP by a second-order cone program. The OCPs of the proposed scheme and its approximation…
In this paper, we investigate the model reference adaptive control approach for uncertain piecewise affine systems with performance guarantees. The proposed approach ensures the error metric, defined as the weighted Euclidean norm of the state tracking error, to be confined within a user-defined time-varying performance bound. We introduce an auxiliary performance function to construct a barrier Lyapunov function. This auxiliary performance signal is reset at each switching instant, which prevents the transgression of the barriers caused by the jumps of the error metric at switching instants. …
A plug-and-play model predictive control (PnP MPC) scheme is proposed for varying-topology networks to track piecewise constant references. The proposed scheme allows subsystems to occasionally join and leave the network while preserving asymptotic stability and recursive feasibility and comprises two main phases. In the redesign phase, passivity-based control is used to ensure that asymptotic stability of the network is preserved. In the transition phase, reconfigurable terminal ingredients are used to ensure that the distributed MPC problem is initially feasible after the PnP operation. The …
This paper proposes a data-driven state feedback controller that enables reference tracking for nonlinear discrete-time systems. The controller is designed based on the identified inverse model of the system and a given reference model, assuming that the identification of the inverse model is carried out using only the system's state/input measurements. When its results are provided, we present conditions that guarantee a certain level of reference tracking performance, regardless of the identification method employed for the inverse model. Specifically, when Gaussian process regression (GPR) …
We investigate a linear quadratic Gaussian (LQG) tracking problem with safety and reachability constraints in the presence of an adversary who mounts an FDI attack on an unknown set of sensors. For each possible set of compromised sensors, we maintain a state estimator disregarding the sensors in that set, and calculate the optimal LQG control input at each time based on this estimate. We propose a control policy which constrains the control input to lie within a fixed distance of the optimal control input corresponding to each state estimate. The control input is obtained at each time step by…
Powered lower-limb exoskeletons provide assistive torques to coordinate limb motion during walking in individuals with movement disorders. Advances in sensing and actuation have improved the wearability and portability of state-of-the-art exoskeletons for walking. Cable-driven exoskeletons offload the actuators away from the user, thus rendering light-weight devices to facilitate locomotion training. However, cable-driven mechanisms experience a slacking behavior if tension is not accurately controlled. Moreover, counteracting forces can arise between the agonist and antagonist motors yielding…
In this paper, we will consider a class of continuous-time stochastic control systems with both unknown nonlinear structure and unknown disturbances, and investigate the capability of the classical proportional-integral-derivative(PID) controller in tracking time-varying reference signals. First, under some suitable conditions on system nonlinear functions, reference signals, and unknown disturbances, we will show that PID controllers can be designed to globally stabilize such systems and ensure the boundedness of the tracking error. Analytic design formulae for PID gain matrices are also prov…
In this paper, we propose a chance constrained stochastic model predictive control scheme for reference tracking of distributed linear time-invariant systems with additive stochastic uncertainty. The chance constraints are reformulated analytically based on mean-variance information, where we design suitable Probabilistic Reachable Sets for constraint tightening. Furthermore, the chance constraints are proven to be satisfied in closed-loop operation. The design of an invariant set for tracking complements the controller and ensures convergence to arbitrary admissible reference points, while a …
Multi-agent systems (MASs) can autonomously learn to solve previously unknown tasks by means of each agent's individual intelligence as well as by collaborating and exploiting collective intelligence. This article considers a group of autonomous agents learning to track the same given reference trajectory in a possibly small number of trials. We propose a novel collective learning control method that combines iterative learning control (ILC) with a collective update strategy. We derive conditions for desirable convergence properties of such systems. We show that the proposed method allows the …
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
In order to autonomously learn to control unknown systems optimally w.r.t. an objective function, Adaptive Dynamic Programming (ADP) is well-suited to adapt controllers based on experience from interaction with the system. In recent years, many researchers focused on the tracking case, where the aim is to follow a desired trajectory. So far, ADP tracking controllers assume that the reference trajectory follows time-invariant exo-system dynamics-an assumption that does not hold for many applications. In order to overcome this limitation, we propose a new Q-function which explicitly incorporates…
In this paper, we present a quasi infinite horizon nonlinear model predictive control (MPC) scheme for tracking of generic reference trajectories. This scheme is applicable to nonlinear systems, which are locally incrementally stabilizable. For such systems, we provide a reference generic offline procedure to compute an incrementally stabilizing feedback with a continuously parameterized quadratic quasi infinite horizon terminal cost. As a result we get a nonlinear reference tracking MPC scheme with a valid terminal cost for general reachable reference trajectories without increasing the onlin…
Reference tracking systems involve a plant that is stabilized by a local feedback controller and a command center that indicates the reference set-point the plant should follow. Typically, these systems are subject to limitations such as disturbances, systems delays, constraints, uncertainties, underperforming controllers, and unmodeled parameters that do not allow them to achieve the desired performance. In situations where it is not possible to redesign the inner-loop system, it is usual to incorporate an outer-loop control that instructs the system to follow a modified reference path such t…
A finite horizon optimal tracking problem is considered for linear dynamical systems subject to parametric uncertainties in the state-space matrices and exogenous disturbances. A suboptimal solution is proposed using a model predictive control (MPC) based explicit dual control approach which enables active uncertainty learning. A novel algorithm for the design of robustly invariant online terminal sets and terminal controllers is presented. Set membership identification is used to update the parameter uncertainty online. A predicted worst-case cost is used in the MPC optimization problem to mo…
