In this paper, we investigate discrete-time decision-making problems in uncertain systems with partially observed states. We consider a non-stochastic model, where uncontrolled disturbances acting on the system take values in bounded sets with unknown distributions. We present a general framework for decision-making in such problems by using the notion of the information state and approximate information state, and introduce conditions to identify an uncertain variable that can be used to compute an optimal strategy through a dynamic program (DP). Next, we relax these conditions and define approximate information states that can be learned from output data without knowledge of system dynamics. We use approximate information states to formulate a DP that yields a strategy with a bounded performance loss. Finally, we illustrate the application of our results in control and reinforcement learning using numerical examples.
University of Tsukuba（筑波大学）；RIKEN AIP（理化学研究所人工智能研究中心）
In the field of reinforcement learning, because of the high cost and risk of policy training in the real world, policies are trained in a simulation environment and transferred to the corresponding real-world environment. However, the simulation environment does not perfectly mimic the real-world environment, lead to model misspecification. Multiple studies report significant deterioration of policy performance in a real-world environment. In this study, we focus on scenarios involving a simulation environment with uncertainty parameters and the set of their possible values, called the uncerta…
Recent advances in deep reinforcement learning have demonstrated the capability of learning complex control policies from many types of environments. When learning policies for safety-critical applications, it is essential to be sensitive to risks and avoid catastrophic events. Towards this goal, we propose an actor-critic framework that models the uncertainty of the future and simultaneously learns a policy based on that uncertainty model. Specifically, given a distribution of the future return for any state and action, we optimize policies for varying levels of conditional Value-at-Risk. The…
Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness
Purdue University（普渡大学）；Carnegie Mellon University（卡内基梅隆大学）；WorldQuant University（WorldQuant大学）；UC Berkeley（加州大学伯克利分校）；Aix-Marseille University（阿维尼翁-马赛大学）；MIT（麻省理工学院）；University of Zurich（苏黎世大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Stuttgart（斯图加特大学）；University of Buenos Aires（布宜诺斯艾利斯大学）；California State University, Fresno（弗雷斯诺加州州立大学）；University of Chicago（芝加哥大学）
In this paper, two types of linear estimators are considered for three related estimation problems involving set-theoretic uncertainty pertaining to $\mathcal{H}_{2}$ and $\mathcal{H}_{\infty}$ balls of frequency-responses. The problems at stake correspond to robust $\mathcal{H}_{2}$ and $\mathcal{H}_{\infty}$ in the face of non-parametric "channel-model" uncertainty and to a nominal $\mathcal{H}_{\infty}$ estimation problem. The estimators considered here are defined by minimizing the worst-case squared estimation error over the "uncertainty set" and by minimizing an average cost under the co…
We introduce the notion of Worst-Case Sensitivity, defined as the worst-case rate of increase in the expected cost of a Distributionally Robust Optimization (DRO) model when the size of the uncertainty set vanishes. We show that worst-case sensitivity is a Generalized Measure of Deviation and that a large class of DRO models are essentially mean-(worst-case) sensitivity problems when uncertainty sets are small, unifying recent results on the relationship between DRO and regularized empirical optimization with worst-case sensitivity playing the role of the regularizer. More generally, DRO solut…
A robust adaptive model predictive control (MPC) algorithm is presented for linear, time invariant systems with unknown dynamics and subject to bounded measurement noise. The system is characterized by an impulse response model, which is assumed to lie within a bounded set called the feasible system set. Online set-membership identification is used to reduce uncertainty in the impulse response. In the MPC scheme, robust constraints are enforced to ensure constraint satisfaction for all the models in the feasible set. The performance objective is formulated as a worst-case cost with respect to …
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Random features is a powerful universal function approximator that inherits the theoretical rigor of kernel methods and can scale up to modern learning tasks. This paper views uncertain system models as unknown or uncertain smooth functions in universal reproducing kernel Hilbert spaces. By directly approximating the one-step dynamics function using random features with uncertain parameters, which are equivalent to a shallow Bayesian neural network, we then view the whole dynamical system as a multi-layer neural network. Exploiting the structure of Hamiltonian dynamics, we show that finding wo…
The dynamics of cellular chemical reactions are variable due to stochastic noise from intrinsic and extrinsic sources. The intrinsic noise is the intracellular fluctuations of molecular copy numbers caused by the probabilistic encounter of molecules and is modeled by the chemical master equation. The extrinsic noise, on the other hand, represents the intercellular variation of the kinetic parameters due to the variation of global factors affecting gene expression. The objective of this paper is to propose a theoretical framework to analyze the combined effect of the intrinsic and the extrinsic…
This paper presents an approach to compute the worst-case gain of the interconnection of a finite time horizon linear time-variant system and a perturbation. The input/output behavior of the uncertainty is described by integral quadratic constraints (IQCs). A condition for the worst-case gain of such an interconnection can be formulated using dissipation theory as a parameterized Riccati differential equation, which depends on the chosen IQC multiplier. A nonlinear optimization problem is formulated to minimize the upper bound of the worst-case gain over a set of admissible IQC multipliers. Th…
This paper outlines a complete methodology for modeling an on-orbit servicing mission scenario and designing a feedback control system for the attitude dynamics that is guaranteed to robustly meet pointing requirements, despite model uncertainties as well as large inertia and flexibility changes throughout the mission scenario. A model of the uncertain plant was derived, which fully captures the dynamics and couplings between all subsystems as well as the decoupled/coupled configurations of the chaser/target system in a single linear fractional representation (LFR). In addition, a new approach…
This paper presents a tool, PELIB, developed in MATLAB /SIMULINK environment to perform pointing performance analysis based on European pointing standards. PELIB is designed as an extension of the Satellite Dynamics Toolbox (SDT), which derives the Linear Fractional Transformation (LFT) models of flexible space structures. The addition of PELIB will allow the users of SDT to perform pointing performance analysis of real mission scenarios in the same environment used for control synthesis. PELIB offers as well the possibility to take into account uncertainties in the system. This feature repres…
This work considers the problem of selecting sensors in a large scale system to minimize the error in estimating its states. More specifically, the state estimation mean-square error(MSE) and worst-case error for Kalman filtering and smoothing. Such selection problems are in general NP-hard, i.e., their solution can only be approximated in practice even for moderately large problems. Due to its low complexity and iterative nature, greedy algorithms are often used to obtain these approximations by selecting one sensor at a time choosing at each step the one that minimizes the estimation perform…
Machine learning algorithms, especially Neural Networks (NNs), are a valuable tool used to approximate non-linear relationships, like the AC-Optimal Power Flow (AC-OPF), with considerable accuracy -- and achieving a speedup of several orders of magnitude when deployed for use. Often in power systems literature, the NNs are trained with a fixed dataset generated prior to the training process. In this paper, we show that adapting the NN training dataset during training can improve the NN performance and substantially reduce its worst-case violations. This paper proposes an algorithm that identif…
