Model-based reinforcement learning is an effective approach for controlling an unknown system. It is based on a longstanding pipeline familiar to the control community in which one performs experiments on the environment to collect a dataset, uses the resulting dataset to identify a model of the system, and finally performs control synthesis using the identified model. As interacting with the system may be costly and time consuming, targeted exploration is crucial for developing an effective control-oriented model with minimal experimentation. Motivated by this challenge, recent work has begun to study finite sample data requirements and sample efficient algorithms for the problem of optimal exploration in model-based reinforcement learning. However, existing theory and algorithms are limited to model classes which are linear in the parameters. Our work instead focuses on models with nonlinear parameter dependencies, and presents the first finite sample analysis of an active learning algorithm suitable for a general class of nonlinear dynamics. In certain settings, the excess control cost of our algorithm achieves the optimal rate, up to logarithmic factors. We validate our approach in simulation, showcasing the advantage of active, control-oriented exploration for controlling nonlinear systems.
While the identification of nonlinear dynamical systems is a fundamental building block of model-based reinforcement learning and feedback control, its sample complexity is only understood for systems that either have discrete states and actions or for systems that can be identified from data generated by i.i.d. random inputs. Nonetheless, many interesting dynamical systems have continuous states and actions and can only be identified through a judicious choice of inputs. Motivated by practical settings, we study a class of nonlinear dynamical systems whose state transitions depend linearly on…
ETH Zurich（苏黎世联邦理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）；University College London（伦敦大学学院）；University of Alberta（阿尔伯塔大学）
Many real-world reinforcement learning tasks require control of complex dynamical systems that involve both costly data acquisition processes and large state spaces. In cases where the transition dynamics can be readily evaluated at specified states (e.g., via a simulator), agents can operate in what is often referred to as planning with a \emph{generative model}. We propose the AE-LSVI algorithm for best-policy identification, a novel variant of the kernelized least-squares value iteration (LSVI) algorithm that combines optimism with pessimism for active exploration (AE). AE-LSVI provably ide…
This paper presents sensitivity analyses of resilience-based active distribution system planning solutions with respect to different parameters. The distribution system planning problem is formulated as a two-stage risk-averse stochastic optimization model with conditional value-at-risk (CVaR) as the risk measure. The probabilistic scenarios are obtained using regional wind profiles, and Monte Carlo simulations are conducted to obtain failure scenarios based on component fragility models. The planning measure includes advanced distribution grid operations with intentional islanding measures. T…
Control-Oriented System Identification: Classical, Learning, and Physics-Informed Approaches
Edwardson School of Industrial Engineering, Purdue University（普渡大学爱德华森工业工程学院）；Electrical and Computer Engineering, University of California San Diego（加州大学圣地亚哥分校电气与计算机工程）；Department of Computer Science, Rice University（莱斯大学计算机科学系）；University of Stuttgart, Institute for Systems Theory and Automatic Control（斯图加特大学系统理论与自动控制研究所）
Active learning is proposed for selection of the next operating points in the design of experiments, for identifying linear parameter-varying systems. We extend existing approaches found in literature to multiple-input multiple-output systems with a multivariate scheduling parameter. Our approach is based on exploiting the probabilistic features of Gaussian process regression to quantify the overall model uncertainty across locally identified models. This results in a flexible framework which accommodates for various techniques to be applied for estimation of local linear models and their corr…
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
We propose an algorithm to actively estimate the parameters of a linear dynamical system. Given complete control over the system's input, our algorithm adaptively chooses the inputs to accelerate estimation. We show a finite time bound quantifying the estimation rate our algorithm attains and prove matching upper and lower bounds which guarantee its asymptotic optimality, up to constants. In addition, we show that this optimal rate is unattainable when using Gaussian noise to excite the system, even with optimally tuned covariance, and analyze several examples where our algorithm provably impr…
Active Sensing and Deferred-Decision Trajectory Optimization for Robust Target Identification
Department of Electrical and Computer Engineering, Purdue University（普渡大学电气与计算机工程系）；Department of Electrical Engineering, University of Notre Dame（诺丁汉大学电气工程系）
Hybrid electric vehicles (HEVs) have an over-actuated system by including two power sources, a battery pack and an internal combustion engine. This feature of HEV is exploited in this paper to simultaneously achieve accurate identification of battery parameters/states. By actively injecting current signals, state of charge, state of health, and other battery parameters can be estimated in a specific sequence to improve the identification performance when compared to the case where all parameters and states are estimated concurrently using the baseline current signals. A dynamic programming str…
Many active noise and vibration control systems require models of the control paths. When the controlled system changes slightly over time, adaptive digital filters for the identification of the models are useful. This paper aims at the investigation of a special class of adaptive digital filters: Orthonormal filter banks possess the robust and simple adaptation of the widely applied Finite Impulse Response (FIR) filters, but at a lower model order, which is important when considering implementation on embedded systems. However, the filter banks require prior knowledge about the resonance freq…
This work proposes an online adaptive identification method for multi-input multi-output (MIMO) switched affine systems with guaranteed parameter convergence. A family of online parameter estimators is used that is equipped with a dual-layer low pass filter architecture to facilitate parameter learning and identification of each subsystem. The filters capture information about the unknown parameters in the form of a prediction error which is used in the parameter estimation algorithm. A salient feature of the proposed method that distinguishes it from most previous results is the use of a memo…
Norwegian University of Science and Technology（挪威科技大学）；SINTEF Digital（SINTEF数字研究所）
The exploding research interest for neural networks in modeling nonlinear dynamical systems is largely explained by the networks' capacity to model complex input-output relations directly from data. However, they typically need vast training data before they can be put to any good use. The data generation process for dynamical systems can be an expensive endeavor both in terms of time and resources. Active learning addresses this shortcoming by acquiring the most informative data, thereby reducing the need to collect enormous datasets. What makes the current work unique is integrating the deep…
With the commitment to climate, globally many countries started reducing brownfield energy production and strongly opting towards green energy resources. However, the optimal allocation of distributed energy resources (DERs) in electrical distribution systems still pertains as a challenging issue to attain the maximum benefits. It happens due to the systems complex behaviour and inappropriate integration of DERs that adversely affects the distribution grid. In this work, we propose a methodology for the optimal allocation of DERs with vulnerable node identification in active electrical distrib…
