This paper deals with subspace estimation in the small sample size regime, where the number of samples is comparable in magnitude with the observation dimension. The traditional estimators, mostly based on the sample correlation matrix, are known to perform well as long as the number of available samples is much larger than the observation dimension. However, in the small sample size regime, the performance degrades. Recently, based on random matrix theory results, a new subspace estimator was introduced, which was shown to be consistent in the asymptotic regime where the number of samples and the observation dimension converge to infinity at the same rate. In practice, this estimator outperforms the traditional ones even for certain scenarios where the observation dimension is small and of the same order of magnitude as the number of samples. In this paper, we address a performance analysis of this recent estimator, by proving a central limit theorem in the above asymptotic regime. We propose an accurate approximation of the mean square error, which can be evaluated numerically.
If the log likelihood is approximately quadratic with constant Hessian, then the maximum likelihood estimator (MLE) is approximately normally distributed. No other assumptions are required. We do not need independent and identically distributed data. We do not need the law of large numbers (LLN) or the central limit theorem (CLT). We do not need sample size going to infinity or anything going to infinity. Presented here is a combination of Le Cam style theory involving local asymptotic normality (LAN) and local asymptotic mixed normality (LAMN) and Cramér style theory involving derivatives and…
In this paper we introduce a general method for estimating the quadratic covariation of one or more spot parameters processes associated with continuous time semimartingales. This estimator is applicable to a wide range of spot parameter processes, and may also be used to estimate the leverage effect of stochastic volatility models. The estimator we introduce is based on sums of squared increments of second differences of the observed process, and the intervals over which the differences are computed are rolling and overlapping. This latter feature lets us take full advantage of the data, and,…
This paper deals with the rate of convergence for the central limit theorem of estimators of the drift coefficient, denoted $θ$, for a Ornstein-Uhlenbeck process $X \coloneqq \{X_t,t\geq0\}$ observed at high frequency. We provide an Approximate minimum contrast estimator and an approximate maximum likelihood estimator of $θ$, namely $\widetildeθ_{n}\coloneqq {1}/{\left(\frac{2}{n} \sum_{i=1}^{n}X_{t_{i}}^{2}\right)}$, and $\widehatθ_{n}\coloneqq -{\sum_{i=1}^{n} X_{t_{i-1}}\left(X_{t_{i}}-X_{t_{i-1}}\right)}/{\left(Δ_{n} \sum_{i=1}^{n} X_{t_{i-1}}^{2}\right)}$, respectively, where $ t_{i} = i …
We propose in this work to derive a CLT in the functional linear regression model to get confidence sets for prediction based on functional linear regression. The main difficulty is due to the fact that estimation of the functional parameter leads to a kind of ill-posed inverse problem. We consider estimators that belong to a large class of regularizing methods and we first show that, contrary to the multivariate case, it is not possible to state a CLT in the topology of the considered functional space. However, we show that we can get a CLT for the weak topology under mild hypotheses and in p…
This paper deals simultaneously with linear structural and functional error-in-variables models (SEIVM and FEIVM), revisiting in this context generalized and modified least squares estimators of the slope and intercept, and some methods of moments estimators of unknown variances of the measurement errors. New joint central limit theorems (CLT's) are established for these estimators in the SEIVM and FEIVM under some first time, so far the most general, respective conditions on the explanatory variables, and under the existence of four moments of the measurement errors. Moreover, due to them bei…
The batch means estimator of the MCMC variance is a simple and effective measure of accuracy for MCMC based ergodic averages. Under various regularity conditions, the estimator has been shown to be consistent for the true variance. However, the estimator can be unstable in practice as it depends directly on the raw MCMC output. A measure of accuracy of the batch means estimator itself, ideally in the form of a confidence interval, is therefore desirable. The asymptotic variance of the batch means estimator is known; however, without any knowledge of asymptotic distribution, asymptotic variance…
CLT and Edgeworth Expansion for m-out-of-n Bootstrap Estimators of The Studentized Median
Department of Industrial Engineering and Management Science Northwestern University（工业工程与管理科学系西北大学）；Department of Computer Science Northwestern University（计算机科学系西北大学）
From a wavelet analysis, one derives a nonparametrical estimator for the spectral density of a Gaussian process with stationary increments. First, the idealistic case of a continuous time path of the process is considered. A punctual Central Limit Theorem (CLT) and an estimation of the Mean Integrate Square Error (MISE) are established. Next, to fit the applications, one considers the case where one observes a path at random times. One built a second estimator obtained by replacing the wavelet coefficients by their discretizations. A second CLT and the corresponding estimation of the MISE are …
We establish limit theorems involving weak convergence of multiple generations of critical and supercritical branching processes. These results arise naturally when dealing with the joint asymptotic behavior of functionals defined in terms of several generations of such processes. Applications of our main result include a functional central limit theorem (CLT), a Darling-Erdös result, and an extremal process result. The limiting process for our functional CLT is an infinite dimensional Brownian motion with sample paths in the infinite product space $(C_0[0,1])^{\infty}$, with the product topol…
Linear structural error-in-variables models with univariate observations are revisited for studying modified least squares estimators of the slope and intercept. New marginal central limit theorems (CLT's) are established for these estimators, assuming the existence of four moments for the measurement errors and that the explanatory variables are in the domain of attraction of the normal law. The latter condition for the explanatory variables is used the first time, and is so far the most general in this context. It is also optimal, or nearly optimal, for our CLT's. Moreover, due to the obtain…
In this paper, we show that the adaptive multidimensional increment ratio estimator of the long range memory parameter defined in Bardet and Dola (2012) satisfies a central limit theorem (CLT in the sequel) for a large semiparametric class of Gaussian fractionally integrated processes with memory parameter $d \in (-0.5,1.25)$. Since the asymptotic variance of this CLT can be computed, tests of stationarity or nonstationarity distinguishing the assumptions $d<0.5$ and $d \geq 0.5$ are constructed. These tests are also consistent tests of unit root. Simulations done on a large benchmark of short…
In this paper, we show that the central limit theorem (CLT) satisfied by the data-driven Multidimensional Increment Ratio (MIR) estimator of the memory parameter d established in Bardet and Dola (2012) for d $\in$ (--0.5, 0.5) can be extended to a semiparametric class of Gaussian fractionally integrated processes with memory parameter d $\in$ (--0.5, 1.25). Since the asymptotic variance of this CLT can be estimated, by data-driven MIR tests for the two cases of stationarity and non-stationarity, so two tests are constructed distinguishing the hypothesis d \textless{} 0.5 and d $\ge$ 0.5, as we…
We propose a Kronecker product model for correlation or covariance matrices in the large dimensional case. The number of parameters of the model increases logarithmically with the dimension of the matrix. We propose a minimum distance (MD) estimator based on a log-linear property of the model, as well as a one-step estimator, which is a one-step approximation to the quasi-maximum likelihood estimator (QMLE). We establish rates of convergence and central limit theorems (CLT) for our estimators in the large dimensional case. A specification test and tools for Kronecker product model selection an…
Samples with a common mean but possibly different, ordered variances arise in various fields such as interlaboratory experiments, field studies or the analysis of sensor data. Estimators for the common mean under ordered variances typically employ random weights, which depend on the sample means and the unbiased variance estimators. They take different forms when the sample estimators are in agreement with the order constraints or not, which complicates even basic analyses such as estimating their variance. We propose to use the jackknife, whose consistency is established for general smooth tw…
Laboratoire de Physique de l’Ecole Normale Supérieure（巴黎高等师范学院物理实验室）；Université PSL（巴黎文理研究大学）；CNRS（法国国家科学研究中心）；Sorbonne Université（索邦大学）；Université Paris-Diderot（巴黎第七大学）；Sorbonne Paris Cité（索邦巴黎西岱大学）；Bocconi University（博科尼大学）
Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$
Computer Science Department, Carnegie Mellon University（卡内基梅隆大学计算机科学系）；Department of Statistics, University of California, Davis（加州大学戴维斯分校统计系）；Department of Statistics, University of Chicago（芝加哥大学统计系）
Virginia Tech（弗吉尼亚理工学院）；Argonne National Laboratory（阿贡国家实验室）；Université Côte d’Azur（蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；LJAD（朱西厄分析、几何与应用实验室）
In recent years, active subspace methods (ASMs) have become a popular means of performing subspace sensitivity analysis on black-box functions. Naively applied, however, ASMs require gradient evaluations of the target function. In the event of noisy, expensive, or stochastic simulators, evaluating gradients via finite differencing may be infeasible. In such cases, often a surrogate model is employed, on which finite differencing is performed. When the surrogate model is a Gaussian process, we show that the ASM estimator is available in closed form, rendering the finite-difference approximation…
We consider a common-components model for multivariate fractional cointegration, in which the $s\geq1$ components have different memory parameters. The cointegrating rank may exceed 1. We decompose the true cointegrating vectors into orthogonal fractional cointegrating subspaces such that vectors from distinct subspaces yield cointegrating errors with distinct memory parameters. We estimate each cointegrating subspace separately, using appropriate sets of eigenvectors of an averaged periodogram matrix of tapered, differenced observations, based on the first $m$ Fourier frequencies, with $m$ fi…
German Aerospace Center (DLR), Institute for AI Safety and Security（德国航空航天中心（DLR）人工智能安全与保护研究所）；German Aerospace Center (DLR), Institute of Frontier Materials on Earth and in Space, Functional, Granular, and Composite Materials（德国航空航天中心（DLR）地球与太空前沿材料研究所功能、颗粒与复合材料部门）；Department of Physics, LMU München（慕尼黑大学物理系）；Karlsruhe Institute of Technology, Institute for Stochastics（卡尔斯鲁厄理工学院概率论研究所）
We propose an extension of the preferential attachment scheme by allowing the connecting probability to depend on time t. We estimate the parameters involved in the model by minimizing the expected squared difference between the number of vertices of degree one and its conditional expectation. The asymptotic properties of the estimators are also investigated when the parameters are time-varying by establishing the central limit theorem (CLT) of the number of vertices of degree one. We propose a new statistic to test whether the parameters have change points. We also offer some methods to estim…
Central Limit Theorems for Stochastic Gradient Descent Quantile Estimators
Department of Statistics, University of Chicago（芝加哥大学统计系）；Department of Statistics and Data Science, Washington University in St. Louis（圣路易斯华盛顿大学统计与数据科学系）
Quantile regression (QR) is becoming increasingly popular due to its relevance in many scientific investigations. However, application of QR can become very challenging when dealing with high-dimensional data, making it necessary to use dimension reduction techniques. Existing dimension reduction techniques focus on the entire conditional distribution. We turn our attention to dimension reduction techniques for conditional quantiles and introduce a method that serves as an intermediate step between linear and nonlinear dimension reduction. The idea is to apply existing linear dimension reducti…
With the scale of data growing every day, reducing the dimensionality (a.k.a. sketching) of high-dimensional data has emerged as a task of paramount importance. Relevant issues to address in this context include the sheer volume of data that may consist of categorical samples, the typically streaming format of acquisition, and the possibly missing entries. To cope with these challenges, the present paper develops a novel categorical subspace learning approach to unravel the latent structure for three prominent categorical (bilinear) models, namely, Probit, Tobit, and Logit. The deterministic P…
Sparse principal component analysis (PCA) involves nonconvex optimization for which the global solution is hard to obtain. To address this issue, one popular approach is convex relaxation. However, such an approach may produce suboptimal estimators due to the relaxation effect. To optimally estimate sparse principal subspaces, we propose a two-stage computational framework named "tighten after relax": Within the 'relax' stage, we approximately solve a convex relaxation of sparse PCA with early stopping to obtain a desired initial estimator; For the 'tighten' stage, we propose a novel algorithm…
High-dimensional streaming data are becoming increasingly ubiquitous in many fields. They often lie in multiple low-dimensional subspaces, and the manifold structures may change abruptly on the time scale due to pattern shift or occurrence of anomalies. However, the problem of detecting the structural changes in a real-time manner has not been well studied. To fill this gap, we propose a dynamic sparse subspace learning approach for online structural change-point detection of high-dimensional streaming data. A novel multiple structural change-point model is proposed and the asymptotic properti…
We consider the problem of subspace estimation in a Bayesian setting. Since we are operating in the Grassmann manifold, the usual approach which consists of minimizing the mean square error (MSE) between the true subspace $U$ and its estimate $\hat{U}$ may not be adequate as the MSE is not the natural metric in the Grassmann manifold. As an alternative, we propose to carry out subspace estimation by minimizing the mean square distance (MSD) between $U$ and its estimate, where the considered distance is a natural metric in the Grassmann manifold, viz. the distance between the projection matrice…
Optimal Personalized Subspace Learning for Multi-view Tensor Observations
Institute for Financial Studies Shandong University（山东大学金融研究院）；School of Statistics and Data Science Southeast University（东南大学统计与数据科学学院）；Department of Statistics and Applied Probability National University of Singapore（新加坡国立大学统计与应用概率系）
Ensemble Conditional Variance Estimation (ECVE) is a novel sufficient dimension reduction (SDR) method in regressions with continuous response and predictors. ECVE applies to general non-additive error regression models. It operates under the assumption that the predictors can be replaced by a lower dimensional projection without loss of information. It is a semiparametric forward regression model based exhaustive sufficient dimension reduction estimation method that is shown to be consistent under mild assumptions. It is shown to outperform central subspace mean average variance estimation (c…
The problem of estimating parameters of switched affine systems with noisy input-output observations is considered. The switched affine models is transformed into a switched linear one by removing its intersection subspace, which is estimated from observations. A subspace technique is proposed to exploit the observations' permutation structure, which transforms the problem of associating observations with subsystems into one of de-permutating a block diagonal matrix, referred as adjacency matrix. Then a normalized spectral clustering algorithm is presented to recover the block structure of adj…
We consider the problem of estimating a signal subspace in the presence of interference that contaminates some proportion of the received observations. Our emphasis is on detecting the contaminated observations so that the signal subspace can be estimated with the contaminated observations discarded. To this end, we employ a signal model which explicitly includes an interference term that is distinct from environmental noise. To detect when the interference term is nonzero, we estimate the interference term using an optimization problem with a sparsity-inducing group SLOPE penalty which accoun…
