We propose in this work to derive a CLT in the functional linear regression model to get confidence sets for prediction based on functional linear regression. The main difficulty is due to the fact that estimation of the functional parameter leads to a kind of ill-posed inverse problem. We consider estimators that belong to a large class of regularizing methods and we first show that, contrary to the multivariate case, it is not possible to state a CLT in the topology of the considered functional space. However, we show that we can get a CLT for the weak topology under mild hypotheses and in particular without assuming any strong assumptions on the decay of the eigenvalues of the covariance operator. Rates of convergence depend on the smoothness of the functional coefficient and on the point in which the prediction is made.
Using Bernstein polynomial approximations, we prove the central limit theorem for linear spectral statistics of sample covariance matrices, indexed by a set of functions with continuous fourth order derivatives on an open interval including $[(1-\sqrt{y})^2,(1+\sqrt{y})^2]$, the support of the Marucenko--Pastur law. We also derive the explicit expressions for asymptotic mean and covariance functions.
In this paper we study the functional central limit theorem for stationary Markov chains with self-adjoint operator and general state space. We investigate the case when the variance of the partial sum is not asymptotically linear in n; and establish that conditional convergence in distribution of partial sums implies functional CLT. The main tools are maximal inequalities that are further exploited to derive conditions for tightness and convergence to the Brownian motion.
Central limit theorems (CLTs) for high-dimensional random vectors with dimension possibly growing with the sample size have received a lot of attention in the recent times. Chernozhukov et al. (2017) proved a Berry--Esseen type result for high-dimensional averages for the class of hyperrectangles and they proved that the rate of convergence can be upper bounded by $n^{-1/6}$ upto a polynomial factor of $\log p$ (where $n$ represents the sample size and $p$ denotes the dimension). Convergence to zero of the bound requires $\log^7p = o(n)$. We improve upon their result which only requires $\log^…
This paper deals simultaneously with linear structural and functional error-in-variables models (SEIVM and FEIVM), revisiting in this context generalized and modified least squares estimators of the slope and intercept, and some methods of moments estimators of unknown variances of the measurement errors. New joint central limit theorems (CLT's) are established for these estimators in the SEIVM and FEIVM under some first time, so far the most general, respective conditions on the explanatory variables, and under the existence of four moments of the measurement errors. Moreover, due to them bei…
We establish limit theorems involving weak convergence of multiple generations of critical and supercritical branching processes. These results arise naturally when dealing with the joint asymptotic behavior of functionals defined in terms of several generations of such processes. Applications of our main result include a functional central limit theorem (CLT), a Darling-Erdös result, and an extremal process result. The limiting process for our functional CLT is an infinite dimensional Brownian motion with sample paths in the infinite product space $(C_0[0,1])^{\infty}$, with the product topol…
An upper bound for the Wasserstein distance is provided in the general framework of the Wiener-Poisson space. Is obtained from this bound a second order Poincaré-type inequality which is useful in terms of computations. For completeness sake, is made a survey of these results on the Wiener space, the Poisson space, and the Wiener-Poisson space, and showed several applications to central limit theorems with relevant examples: linear functionals of Gaussian subordinated field (where the subordinated field can be processes like fractional Brownian motion or the solution of the Ornstein-Uhlenbeck …
Random Fisher matrices arise naturally in multivariate statistical analysis and understanding the properties of its eigenvalues is of primary importance for many hypothesis testing problems like testing the equality between two multivariate population covariance matrices, or testing the independence between sub-groups of a multivariate random vector. This paper is concerned with the properties of a large-dimensional Fisher matrix when the dimension of the population is proportionally large compared to the sample size. Most of existing works on Fisher matrices deal with a particular Fisher matr…
This paper studies the fluctuations of the signal-to-noise ratio (SNR) of minimum variance distorsionless response (MVDR) filters implementing diagonal loading in the estimation of the covariance matrix. Previous results in the signal processing literature are generalized and extended by considering both spatially as well as temporarily correlated samples. Specifically, a central limit theorem (CLT) is established for the fluctuations of the SNR of the diagonally loaded MVDR filter, under both supervised and unsupervised training settings in adaptive filtering applications. Our second-order an…
Statistical Inference for Gradient Boosting Regression
School of Economics, Fudan University（复旦大学经济学院）；Department of Statistics and Data Science, The Wharton School, University of Pennsylvania（宾夕法尼亚大学沃顿商学院统计与数据科学系）
The augmented GARCH model is a unification of numerous extensions of the popular and widely used ARCH process. It was introduced by Duan and besides ordinary (linear) GARCH processes, it contains exponential GARCH, power GARCH, threshold GARCH, asymmetric GARCH, etc. In this paper, we study the probabilistic structure of augmented $\mathrm {GARCH}(1,1)$ sequences and the asymptotic distribution of various functionals of the process occurring in problems of statistical inference. Instead of using the Markov structure of the model and implied mixing properties, we utilize independence properties…
Functional linear regression analysis aims to model regression relations which include a functional predictor. The analog of the regression parameter vector or matrix in conventional multivariate or multiple-response linear regression models is a regression parameter function in one or two arguments. If, in addition, one has scalar predictors, as is often the case in applications to longitudinal studies, the question arises how to incorporate these into a functional regression model. We study a varying-coefficient approach where the scalar covariates are modeled as additional arguments of the …
We investigate invariant random fields on the sphere using a new type of spherical wavelets, called needlets. These are compactly supported in frequency and enjoy excellent localization properties in real space, with quasi-exponentially decaying tails. We show that, for random fields on the sphere, the needlet coefficients are asymptotically uncorrelated for any fixed angular distance. This property is used to derive CLT and functional CLT convergence results for polynomial functionals of the needlet coefficients: here the asymptotic theory is considered in the high-frequency sense. Our propos…
In this study, we develop an asymptotic theory of nonparametric regression for a locally stationary functional time series. First, we introduce the notion of a locally stationary functional time series (LSFTS) that takes values in a semi-metric space. Then, we propose a nonparametric model for LSFTS with a regression function that changes smoothly over time. We establish the uniform convergence rates of a class of kernel estimators, the Nadaraya-Watson (NW) estimator of the regression function, and a central limit theorem of the NW estimator.
We study regression models for the situation where both dependent and independent variables are square-integrable stochastic processes. Questions concerning the definition and existence of the corresponding functional linear regression models and some basic properties are explored for this situation. We derive a representation of the regression parameter function in terms of the canonical components of the processes involved. This representation establishes a connection between functional regression and functional canonical analysis and suggests alternative approaches for the implementation of…
We consider a spatial functional linear regression, where a scalar response is related to a square integrable spatial functional process. We use a smoothing spline estimator for the functional slope parameter and establish a finite sample bound for variance of this estimator under mixing spatial dependence. Then, we give a bound of the prediction error. Finally, we illustrate our results by simulations
We tackle estimation and prediction at non-visted sites in a spatial semi-functional linear regression model with derivatives that combines a functional linear model with a nonparametric regression one. The parametric part is estimated by a method of moments and the other one by a local linear estimator. We establish the convergence rate of the resulting estimators and predictor. A simulation study and an application to ozone pollution prediction at non-visted sites are proposed to illustrate our results.
We introduce a new model of linear regression for random functional inputs taking into account the first order derivative of the data. We propose an estimation method which comes down to solving a special linear inverse problem. Our procedure tackles the problem through a double and synchronized penalization. An asymptotic expansion of the mean square prevision error is given. The model and the method are applied to a benchmark dataset of spectrometric curves and compared with other functional models.
We propose a novel test statistic for testing exogeneity in the functional linear regression model. In contrast to Hausman-type tests in finite dimensional linear regression setups, a direct extension to the functional linear regression model is not possible. Instead, we propose a test statistic based on the sum of the squared difference of projections of the two estimators for testing the null hypothesis of exogeneity in the functional linear regression model. We derive asymptotic normality under the null and consistency under general alternatives. Moreover, we prove bootstrap consistency res…
The main purpose is to estimate the regression function of a real random variable with functional explanatory variable by using a recursive nonparametric kernel approach. The mean square error and the almost sure convergence of a family of recursive kernel estimates of the regression function are derived. These results are established with rates and precise evaluation of the constant terms. Also, a central limit theorem for this class of estimators is established. The method is evaluated on simulations and real data set studies.
We consider a strictly stationary and ergodic sequence of random elements taking values in some Hilbert space. Our target is to study the weak convergence of the discrete Fourier transforms under sharp conditions. As a side-result we obtain the regular CLT for partial sums under mild assumptions.
We study a non linear regression model with functional data as inputs and scalar response. We propose a pointwise estimate of the regression function that maps a Hilbert space onto the real line by a local linear method. We provide the asymptotic mean square error. Computations involve a linear inverse problem as well as a representation of the small ball probability of the data and are based on recent advances in this area. The rate of convergence of our estimate outperforms those already obtained in the literature on this model.
CLT-Optimal Parameter Error Bounds for Linear System Identification
Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, California, USA（明希电气与计算机工程系，南加州大学，洛杉矶，加利福尼亚州，美国）
Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$
Computer Science Department, Carnegie Mellon University（卡内基梅隆大学计算机科学系）；Department of Statistics, University of California, Davis（加州大学戴维斯分校统计系）；Department of Statistics, University of Chicago（芝加哥大学统计系）
In this paper, we study statistical inference in functional quantile regression for scalar response and a functional covariate. Specifically, we consider a functional linear quantile regression model where the effect of the covariate on the quantile of the response is modeled through the inner product between the functional covariate and an unknown smooth regression parameter function that varies with the level of quantile. The objective is to test that the regression parameter is constant across several quantile levels of interest. The parameter function is estimated by combining ideas from f…
Regressing a scalar response on a random function is nowadays a common situation. In the nonparametric setting, this paper paves the way for making the local linear regression based on a projection approach a prominent method for solving this regression problem. Our asymptotic results demonstrate that the functional local linear regression outperforms its functional local constant counterpart. Beyond the estimation of the regression operator itself, the local linear regression is also a useful tool for predicting the functional derivative of the regression operator, a promising mathematical ob…
We consider nonparametric regression in the context of functional data, that is, when a random sample of functions is observed on a fine grid. We obtain a functional asymptotic normality result allowing to build simultaneous confidence bands (SCB) for various estimation and inference tasks. Two applications to a SCB procedure for the regression function and to a goodness-of-fit test for curvilinear regression models are proposed. The first one has improved accuracy upon the other available methods while the second can detect local departures from a parametric shape, as opposed to the usual goo…
We study a functional linear regression model that deals with functional responses and allows for both functional covariates and high-dimensional vector covariates. The proposed model is flexible and nests several functional regression models in the literature as special cases. Based on the theory of reproducing kernel Hilbert spaces (RKHS), we propose a penalized least squares estimator that can accommodate functional variables observed on discrete sample points. Besides a conventional smoothness penalty, a group Lasso-type penalty is further imposed to induce sparsity in the high-dimensional…
Shape restrictions on functional regression coefficients such as non-negativity, monotonicity, convexity or concavity are often available in the form of a prior knowledge or required to maintain a structural consistency in functional regression models. A new estimation method is developed in shape-constrained functional regression models using Bernstein polynomials. Specifically, estimation approaches from nonparametric regression are extended to functional data, properly accounting for shape-constraints in a large class of functional regression models such as scalar-on-function regression (SO…
Consider a bipartite network where $N$ consumers choose to buy or not to buy $M$ different products. This paper considers the properties of the logistic regression of the $N\times M$ array of i-buys-j purchase decisions, $\left[Y_{ij}\right]_{1\leq i\leq N,1\leq j\leq M}$, onto known functions of consumer and product attributes under asymptotic sequences where (i) both $N$ and $M$ grow large and (ii) the average number of products purchased per consumer is finite in the limit. This latter assumption implies that the network of purchases is sparse: only a (very) small fraction of all possible p…
Functional linear regression is a useful extension of simple linear regression and has been investigated by many researchers. However, functional variable selection problems when multiple functional observations exist, which is the counterpart in the functional context of multiple linear regression, is seldom studied. Here we propose a method using group smoothly clipped absolute deviation penalty (gSCAD) which can perform regression estimation and variable selection simultaneously. We show the method can identify the true model consistently and discuss construction of pointwise confidence int…
The R add-on package FDboost is a flexible toolbox for the estimation of functional regression models by model-based boosting. It provides the possibility to fit regression models for scalar and functional response with effects of scalar as well as functional covariates, i.e., scalar-on-function, function-on-scalar and function-on-function regression models. In addition to mean regression, quantile regression models as well as generalized additive models for location scale and shape can be fitted with FDboost. Furthermore, boosting can be used in high-dimensional data settings with more covari…
In this paper, we apply empirical likelihood method to inference for the regression parameters in the partial functional linear regression models based on B spline. We prove that the empirical log likelihood ratio for the regression parameters converges in law to a weighted sum of independent chi square distributions and run simulations to assess the finite sample performance of our method.
We investigate nonparametric regression methods based on spatial depth and quantiles when the response and the covariate are both functions. As in classical quantile regression for finite dimensional data, regression techniques developed here provide insight into the influence of the functional covariate on different parts, like the center as well as the tails, of the conditional distribution of the functional response. Depth and quantile based nonparametric regressions are useful to detect heteroscedasticity in functional regression. We derive the asymptotic behaviour of nonparametric depth a…
A characteristic feature of functional data is the presence of phase variability in addition to amplitude variability. Existing functional regression methods do not handle time variability in an explicit and efficient way. In this paper we introduce a functional regression method that incorporates time warping as an intrinsic part of the model. The method achieves good predictive power in a parsimonious way and allows unified statistical inference about phase and amplitude components. The asymptotic distribution of the estimators is derived and the finite-sample properties are studied by simul…
In the functional linear regression model, many methods have been proposed and studied to estimate the slope function while the functional predictor was observed in the entire domain. However, works on functional linear regression models with partially observed trajectories have received less attention. In this paper, to fill the literature gap we consider the scenario where individual functional predictor may be observed only on part of the domain. Depending on whether measurement error is presented in functional predictors, two methods are developed, one is based on linear functionals of the…
We propose a prediction procedure for the functional linear quantile regression model by using partial quantile covariance techniques and develop a simple partial quantile regression (SIMPQR) algorithm to efficiently extract partial quantile regression (PQR) basis for estimating functional coefficients. We further extend our partial quantile covariance techniques to functional composite quantile regression (CQR) defining partial composite quantile covariance. There are three major contributions. (1) We define partial quantile covariance between two scalar variables through linear quantile regr…
We place ourselves in a functional regression setting and propose a novel methodology for regressing a real output on vector-valued functional covariates. This methodology is based on the notion of signature, which is a representation of a function as an infinite series of its iterated integrals. The signature depends crucially on a truncation parameter for which an estimator is provided, together with theoretical guarantees. An empirical study on both simulated and real-world datasets shows that the resulting methodology is competitive with traditional functional linear models, in particular …
In this paper, a functional partial quantile regression approach, a quantile regression analog of the functional partial least squares regression, is proposed to estimate the function-on-function linear quantile regression model. A partial quantile covariance function is first used to extract the functional partial quantile regression basis functions. The extracted basis functions are then used to obtain the functional partial quantile regression components and estimate the final model. In our proposal, the functional forms of the discretely observed random variables are first constructed via …
In this paper we prove large deviations principles for the Nadaraya-Watson estimator of the regression of a real-valued variable with a functional covariate. Under suitable conditions, we show pointwise and uniform large deviations theorems with good rate functions.
We consider the recursive estimation of a regression functional where the explanatory variables take values in some functional space. We prove the almost sure convergence of such estimates for dependent functional data. Also we derive the mean quadratic error of the considered class of estimators. Our results are established with rates and asymptotic appear bounds, under strong mixing condition.
Functional data analysis is a growing research field as more and more practical applications involve functional data. In this paper, we focus on the problem of regression and classification with functional predictors: the model suggested combines an efficient dimension reduction procedure [functional sliced inverse regression, first introduced by Ferré & Yao (Statistics, 37, 2003, 475)], for which we give a regularized version, with the accuracy of a neural network. Some consistency results are given and the method is successfully confronted to real-life data.
We study the non-parametric isotonic regression problem for bivariate elicitable functionals that are given as an elicitable univariate functional and its Bayes risk. Prominent examples for functionals of this type are (mean, variance) and (Value-at-Risk, Expected Shortfall), where the latter pair consists of important risk measures in finance. We present our results for totally ordered covariates but extenstions to partial orders are given in the appendix.
We consider nonparametric functional regression when both predictors and responses are functions. More specifically, we let $(X_1,Y_1),...,(X_n,Y_n)$ be random elements in $\mathcal{F}\times\mathcal{H}$ where $\mathcal{F}$ is a semi-metric space and $\mathcal{H}$ is a separable Hilbert space. Based on a recently introduced notion of weak dependence for functional data, we showed the almost sure convergence rates of both the Nadaraya-Watson estimator and the nearest neighbor estimator, in a unified manner. Several factors, including functional nature of the responses, the assumptions on the fun…
