In this paper, we consider the problem of computing parameters of an objective function for a discrete-time optimal control problem from state and control trajectories with active control constraints. We propose a novel method of inverse optimal control that has a computationally efficient online form in which pairs of states and controls from given state and control trajectories are processed sequentially without being stored or processed in batches. We establish conditions guaranteeing the uniqueness of the objective-function parameters computed by our proposed method from trajectories with active control constraints. We illustrate our proposed method in simulation.
This paper addresses the problem of online inverse reinforcement learning for nonlinear systems with modeling uncertainties while in the presence of unknown disturbances. The developed approach observes state and input trajectories for an agent and identifies the unknown reward function online. Sub-optimality introduced in the observed trajectories by the unknown external disturbance is compensated for using a novel model-based inverse reinforcement learning approach. The observer estimates the external disturbances and uses the resulting estimates to learn the dynamic model of the demonstrato…
We address the problem of simultaneously learning and control in an online receding horizon control setting. We consider the control of an unknown linear dynamical system with general cost functions and affine constraints on the control input. Our goal is to develop an online learning algorithm that minimizes the dynamic regret, which is defined as the difference between the cumulative cost incurred by the algorithm and that of the best policy with full knowledge of the system, cost functions and state and that satisfies the control input constraints. We propose a novel approach to explore in …
This paper addresses the problem of online inverse reinforcement learning for systems with limited data and uncertain dynamics. In the developed approach, the state and control trajectories are recorded online by observing an agent perform a task, and reward function estimation is performed in real-time using a novel inverse reinforcement learning approach. Parameter estimation is performed concurrently to help compensate for uncertainties in the agent's dynamics. Data insufficiency is resolved by developing a data-driven update law to estimate the optimal feedback controller. The estimated co…
In this paper we provide provable regret guarantees for an online meta-learning receding horizon control algorithm in an iterative control setting. We consider the setting where, in each iteration the system to be controlled is a linear deterministic system that is different and unknown, the cost for the controller in an iteration is a general additive cost function and there are affine control input constraints. By analysing conditions under which sub-linear regret is achievable, we prove that the meta-learning online receding horizon controller achieves an average of the dynamic regret for t…
The inverse linear-quadratic optimal control problem is a system identification problem whose aim is to recover the quadratic cost function and hence the closed-loop system matrices based on observations of optimal trajectories. In this paper, the discrete-time, finite-horizon case is considered, where the agents are also assumed to be homogeneous and indistinguishable. The latter means that the agents all have the same dynamics and objective functions and the observations are in terms of "snap shots" of all agents at different time instants, but what is not known is "which agent moved where" …
We develop a one-Newton-step-per-horizon, online, lag-$L$, model predictive control (MPC) algorithm for solving discrete-time, equality-constrained, nonlinear dynamic programs. Based on recent sensitivity analysis results for the target problems class, we prove that the approach exhibits a behavior that we call superconvergence; that is, the tracking error with respect to the full horizon solution is not only stable for successive horizon shifts, but also decreases with increasing shift order to a minimum value that decays exponentially in the length of the receding horizon. The key analytical…
The goal of Inverse Optimal Control (IOC) is to identify the underlying objective function based on observed optimal trajectories. It provides a powerful framework to model expert's behavior, and a data-driven way to design an objective function so that the induced optimal control is adapted to a contextual environment. In this paper, we design an IOC algorithm for linear-quadratic tracking problems with random time horizon, and prove the statistical consistency of the algorithm. More specifically, the proposed estimator is the solution to a convex optimization problem, which means that the es…
This paper considers online optimal control with affine constraints on the states and actions under linear dynamics with bounded random disturbances. The system dynamics and constraints are assumed to be known and time-invariant but the convex stage cost functions change adversarially. To solve this problem, we propose Online Gradient Descent with Buffer Zones (OGD-BZ). Theoretically, we show that OGD-BZ with proper parameters can guarantee the system to satisfy all the constraints despite any admissible disturbances. Further, we investigate the policy regret of OGD-BZ, which compares OGD-BZ's…
Decentralized Online Learning for Random Inverse Problems Over Graphs
No.2 High School of East China Normal University（华东师范大学第二附属中学）；School of Mathematical Sciences, East China Normal University（华东师范大学数学科学学院）；Key Laboratory of Management, Decision and Information Systems, Institute of Systems Science, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院系统科学研究所管理、决策与信息系统重点实验室）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Human drivers have limited and time-varying cognitive resources when making decisions in real-world traffic scenarios, which often leads to unique and stochastic behaviors that can not be explained by perfect rationality assumption, a widely accepted premise in modeling driving behaviors that presume drivers rationally make decisions to maximize their own rewards under all circumstances. To explicitly address this disadvantage, this study presents a novel driver behavior model that aims to capture the resource rationality and stochasticity of the human driver's behaviors in realistic longitudi…
This paper proposes a primal-dual framework to learn a stable estimator for linear constrained estimation problems leveraging the moving horizon approach. To avoid the online computational burden in most existing methods, we learn a parameterized function offline to approximate the primal estimate. Meanwhile, a dual estimator is trained to check the suboptimality of the primal estimator during execution time. Both the primal and dual estimators are learned from data using supervised learning techniques, and the explicit sample size is provided, which enables us to guarantee the quality of each…
CBF-QP safety filters are pointwise minimizers of the control effort at a given state vector, i.e., myopically optimal at each time instant. But are they optimal over the entire infinite time horizon? What does it even mean for a controlled dynamic systems to be "optimally safe" as opposed to, conventionally "optimally stable"? When disturbances, deterministic and stochastic, have unknown upper bounds, how should safety be defined to allow a graceful degradation under disturbances? Can safety filters be designed to guarantee such weaker safety properties as well as the optimality of safety ove…
This paper introduces a data-based moving horizon estimation (MHE) scheme for linear time-invariant discrete-time systems. The scheme solely relies on collected data without employing any system identification step. Robust global exponential stability of the data-based MHE is proven under standard assumptions for the case where the online output measurements are corrupted by some non-vanishing measurement noise. A simulation example illustrates the behavior of the data-based MHE scheme.
