Recent progress in reinforcement learning has led to remarkable performance in a range of applications, but its deployment in high-stakes settings remains quite rare. One reason is a limited understanding of the behavior of reinforcement algorithms, both in terms of their regret and their ability to learn the underlying system dynamics---existing work is focused almost exclusively on characterizing rates, with little attention paid to the constants multiplying those rates that can be critically important in practice. To start to address this challenge, we study perhaps the simplest non-bandit reinforcement learning problem: linear quadratic adaptive control (LQAC). By carefully combining recent finite-sample performance bounds for the LQAC problem with a particular (less-recent) martingale central limit theorem, we are able to derive asymptotically-exact expressions for the regret, estimation error, and prediction error of a rate-optimal stepwise-updating LQAC algorithm. In simulations on both stable and unstable systems, we find that our asymptotic theory also describes the algorithm's finite-sample behavior remarkably well.
We study local asymptotic properties of likelihood ratios of certain Heston models. We distinguish three cases: subcritical, critical and supercritical models. For the drift parameters, local asymptotic normality is proved in the subcritical case, only local asymptotic quadraticity is shown in the critical case, while in the supercritical case not even local asymptotic quadraticity holds. For certain submodels, local asymptotic normality is proved in the critical case, and local asymptotic mixed normality is shown in the supercritical case. As a consequence, asymptotically optimal (randomized)…
This paper considers adaptive, minimax estimation of a quadratic functional in a nonparametric instrumental variables (NPIV) model, which is an important problem in optimal estimation of a nonlinear functional of an ill-posed inverse regression with an unknown operator. We first show that a leave-one-out, sieve NPIV estimator of the quadratic functional can attain a convergence rate that coincides with the lower bound previously derived in Chen and Christensen [2018]. The minimax rate is achieved by the optimal choice of the sieve dimension (a key tuning parameter) that depends on the smoothne…
In this paper we study the asymptotic theory for quadratic variation of a harmonizable fractional $\al$-stable process. We show a law of large numbers with a non-ergodic limit and obtain weak convergence towards a Lévy-driven Rosenblatt random variable when the Hurst parameter satisfies $H\in (1/2,1)$ and $\al(1-H)<1/2$. This result complements the asymptotic theory for fractional stable processes investigated in e.g. \cite{BHP19,BLP17,BP17,BPT20,LP18,MOP20}.
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper (2007) for estimation of unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk. It means that the asymptotic quadratic risk for this procedure coincides with a sharp lower bound.
The article is devoted to the nonparametric estimation of the quadratic covariation of non-synchronously observed Itô processes in an additive microstructure noise model. In a high-frequency setting, we aim at establishing an asymptotic distribution theory for a generalized multiscale estimator including a feasible central limit theorem with optimal convergence rate on convenient regularity assumptions. The inevitably remaining impact of asynchronous deterministic sampling schemes and noise corruption on the asymptotic distribution is precisely elucidated. A case study for various important ex…
We study the problem of adaptive control in partially observable linear quadratic Gaussian control systems, where the model dynamics are unknown a priori. We propose LqgOpt, a novel reinforcement learning algorithm based on the principle of optimism in the face of uncertainty, to effectively minimize the overall control cost. We employ the predictor state evolution representation of the system dynamics and deploy a recently proposed closed-loop system identification method, estimation, and confidence bound construction. LqgOpt efficiently explores the system dynamics, estimates the model param…
We prove conditional asymptotic normality of a class of quadratic U-statistics that are dominated by their degenerate second order part and have kernels that change with the number of observations. These statistics arise in the construction of estimators in high-dimensional semi- and non-parametric models, and in the construction of nonparametric confidence sets. This is illustrated by estimation of the integral of a square of a density or regression function, and estimation of the mean response with missing data. We show that estimators are asymptotically normal even in the case that the rate…
A popular Bayesian nonparametric approach to survival analysis consists in modeling hazard rates as kernel mixtures driven by a completely random measure. In this paper we derive asymptotic results for linear and quadratic functionals of such random hazard rates. In particular, we prove central limit theorems for the cumulative hazard function and for the path-second moment and path-variance of the hazard rate. Our techniques are based on recently established criteria for the weak convergence of single and double stochastic integrals with respect to Poisson random measures. We illustrate our r…
An adaptive nonparametric estimation procedure is constructed for the estimation problem of heteroscedastic regression when the noise variance depends on the unknown regression. A non-asymptotic upper bound for a quadratic risk (an oracle inequality) is constructed.
Texas A & M University（德克萨斯农工大学）；Indian Institute of Science（印度科学学院）
We consider the problem of controlling an unknown stochastic linear system with quadratic costs - called the adaptive LQ control problem. We re-examine an approach called ''Reward Biased Maximum Likelihood Estimate'' (RBMLE) that was proposed more than forty years ago, and which predates the ''Upper Confidence Bound'' (UCB) method as well as the definition of ''regret'' for bandit problems. It simply added a term favoring parameters with larger rewards to the criterion for parameter estimation. We show how the RBMLE and UCB methods can be reconciled, and thereby propose an Augmented RBMLE-UCB …
A wide literature is available on the asymptotic behavior of the Durbin-Watson statistic for autoregressive models. However, it is impossible to find results on the Durbin-Watson statistic for autoregressive models with adaptive control. Our purpose is to fill the gap by establishing the asymptotic behavior of the Durbin Watson statistic for ARX models in adaptive tracking. On the one hand, we show the almost sure convergence as well as the asymptotic normality of the least squares estimators of the unknown parameters of the ARX models. On the other hand, we establish the almost sure convergen…
We build confidence balls for the common density $s$ of a real valued sample $X_1,...,X_n$. We use resampling methods to estimate the projection of $s$ onto finite dimensional linear spaces and a model selection procedure to choose an optimal approximation space. The covering property is ensured for all $n\geq2$ and the balls are adaptive over a collection of linear spaces.
We consider estimation of the quadratic (co)variation of a semimartingale from discrete observations which are irregularly spaced under high-frequency asymptotics. In the univariate setting, results by Jacod (2008) are generalized to the case of irregular observations. In the two-dimensional setup under non-synchronous observations, we derive a stable central limit theorem for the Hayashi-Yoshida estimator in the presence of jumps. We reveal how idiosyncratic and simultaneous jumps affect the asymptotic distribution. Observation times generated by Poisson processes are explicitly discussed.
Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
Department of Industrial Engineering and Operations Research, Columbia University（工业工程与运营管理系，哥伦比亚大学）；Data Science Institute, Columbia University（数据科学研究所，哥伦比亚大学）
针对连续时间随机 LQ 控制的无模型 RL，本文提出由 critic 与 actor 自适应调节熵正则化和策略方差的数据驱动探索机制，以少量调参实现匹配已知最优固定调度结果的次线性遗憾，并在实验中加速收敛、降低遗憾。
Reinforcement learning for adaptive interior point methods in convex quadratic programming
Department of Aerospace Engineering, Institute of Applied Mathematics and Scientific Computing, University of the Bundeswehr Munich（航空航天工程系、应用数学与科学计算研究所、联邦国防军大学穆尔尼奇分校）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；SEAS Harvard University（哈佛大学工程与应用科学学院）
We study the adaptive control of an unknown linear system with a quadratic cost function subject to safety constraints on both the states and actions. The challenges of this problem arise from the tension among safety, exploration, performance, and computation. To address these challenges, we propose a polynomial-time algorithm that guarantees feasibility and constraint satisfaction with high probability under proper conditions. Our algorithm is implemented on a single trajectory and does not require system restarts. Further, we analyze the regret of our learning algorithm compared to the opti…
We consider adaptive control of the Linear Quadratic Regulator (LQR), where an unknown linear system is controlled subject to quadratic costs. Leveraging recent developments in the estimation of linear systems and in robust controller synthesis, we present the first provably polynomial time algorithm that provides high probability guarantees of sub-linear regret on this problem. We further study the interplay between regret minimization and parameter estimation by proving a lower bound on the expected regret in terms of the exploration schedule used by any algorithm. Finally, we conduct a nume…
Massachusetts Institute of Technology（麻省理工学院）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Establishing a fast rate of convergence for optimization methods is crucial to their applicability in practice. With the increasing popularity of deep learning over the past decade, stochastic gradient descent and its adaptive variants (e.g. Adagrad, Adam, etc.) have become prominent methods of choice for machine learning practitioners. While a large number of works have demonstrated that these first order optimization methods can achieve sub-linear or linear convergence, we establish local quadratic convergence for stochastic gradient descent with adaptive step size for problems such as matri…
We consider the problem of robust and adaptive model predictive control (MPC) of a linear system, with unknown parameters that are learned along the way (adaptive), in a critical setting where failures must be prevented (robust). This problem has been studied from different perspectives by different communities. However, the existing theory deals only with the case of quadratic costs (the LQ problem), which limits applications to stabilisation and tracking tasks only. In order to handle more general (non-convex) costs that naturally arise in many practical problems, we carefully select and bri…
National University of Defense Technology（国防科技大学）；Tsinghua University（清华大学）；University of Utah（犹他大学）
Heavy ball momentum is crucial in accelerating (stochastic) gradient-based optimization algorithms for machine learning. Existing heavy ball momentum is usually weighted by a uniform hyperparameter, which relies on excessive tuning. Moreover, the calibrated fixed hyperparameter may not lead to optimal performance. In this paper, to eliminate the effort for tuning the momentum-related hyperparameter, we propose a new adaptive momentum inspired by the optimal choice of the heavy ball momentum for quadratic optimization. Our proposed adaptive heavy ball momentum can improve stochastic gradient de…
Guangdong Ocean University（广东海洋大学）；Southeast University（东南大学）；University of Leicester（莱斯特大学）
The time-varying quadratic miniaturization (TVQM) problem, as a hotspot currently, urgently demands a more reliable and faster--solving model. To this end, a novel adaptive coefficient constructs framework is presented and realized to improve the performance of the solution model, leading to the adaptive zeroing-type neural dynamics (AZTND) model. Then the AZTND model is applied to solve the TVQM problem. The adaptive coefficients can adjust the step size of the model online so that the solution model converges faster. At the same time, the integration term develops to enhance the robustness o…
Indian Institute of Science（印度科学研究院）；Wayne State University（韦恩州立大学）；The Ohio State University（俄亥俄州立大学）
In this paper, we study the problem of regret minimization in reinforcement learning (RL) under differential privacy constraints. This work is motivated by the wide range of RL applications for providing personalized service, where privacy concerns are becoming paramount. In contrast to previous works, we take the first step towards non-tabular RL settings, while providing a rigorous privacy guarantee. In particular, we consider the adaptive control of differentially private linear quadratic (LQ) systems. We develop the first private RL algorithm, PRL, which is able to attain a sub-linear regr…
We consider least-squares problems with quadratic regularization and propose novel sketching-based iterative methods with an adaptive sketch size. The sketch size can be as small as the effective dimension of the data matrix to guarantee linear convergence. However, a major difficulty in choosing the sketch size in terms of the effective dimension lies in the fact that the latter is usually unknown in practice. Current sketching-based solvers for regularized least-squares fall short on addressing this issue. Our main contribution is to propose adaptive versions of standard sketching-based iter…
Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系）
We propose a randomized algorithm with quadratic convergence rate for convex optimization problems with a self-concordant, composite, strongly convex objective function. Our method is based on performing an approximate Newton step using a random projection of the Hessian. Our first contribution is to show that, at each iteration, the embedding dimension (or sketch size) can be as small as the effective dimension of the Hessian matrix. Leveraging this novel fundamental result, we design an algorithm with a sketch size proportional to the effective dimension and which exhibits a quadratic rate o…
California Institute of Technology（加州理工学院）；Purdue University（普渡大学）
Thompson Sampling (TS) is an efficient method for decision-making under uncertainty, where an action is sampled from a carefully prescribed distribution which is updated based on the observed data. In this work, we study the problem of adaptive control of stabilizable linear-quadratic regulators (LQRs) using TS, where the system dynamics are unknown. Previous works have established that $\tilde O(\sqrt{T})$ frequentist regret is optimal for the adaptive control of LQRs. However, the existing methods either work only in restrictive settings, require a priori known stabilizing controllers, or ut…
SubQuad: Near-Quadratic-Free Structure Inference with Distribution-Balanced Objectives in Adaptive Receptor framework
University of Macau（澳门大学）；University of Pennsylvania（宾夕法尼亚大学）；University of Southampton（南安普顿大学）；Juntendo University（顺天堂大学）；University of Bologna（博洛尼亚大学）
