In a recent paper Yang and Stufken [Ann. Statist. 40 (2012a) 1665-1685] gave sufficient conditions for complete classes of designs for nonlinear regression models. In this note we demonstrate that there is an alternative way to validate this result. Our main argument utilizes the fact that boundary points of moment spaces generated by Chebyshev systems possess unique representations.
We analyse the prediction error of principal component regression (PCR) and prove non-asymptotic upper bounds for the corresponding squared risk. Under mild assumptions, we show that PCR performs as well as the oracle method obtained by replacing empirical principal components by their population counterparts. Our approach relies on upper bounds for the excess risk of principal component analysis.
Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We derive an explicit finite-sample risk bound for regularization-based estimators that simultaneously accounts for (i) the structure of the ambient function space, (ii) the regularity of the true regression…
Principal Components Regression (PCR) is a traditional tool for dimension reduction in linear regression that has been both criticized and defended. One concern about PCR is that obtaining the leading principal components tends to be computationally demanding for large data sets. While random projections do not possess the optimality properties of the leading principal subspace, they are computationally appealing and hence have become increasingly popular in recent years. In this paper, we present an analysis showing that for random projections satisfying a Johnson-Lindenstrauss embedding prop…
Principal loading analysis is a dimension reduction method that discards variables which have only a small distorting effect on the covariance matrix. As a special case, principal loading analysis discards variables that are not correlated with the remaining ones. In multivariate linear regression on the other hand, predictors that are neither correlated with both the remaining predictors nor with the dependent variables have a regression coefficients equal to zero. Hence, if the goal is to select a number of predictors, variables that do not correlate are discarded as it is also done in princ…
In scientific applications, multivariate observations often come in tandem with temporal or spatial covariates, with which the underlying signals vary smoothly. The standard approaches such as principal component analysis and factor analysis neglect the smoothness of the data, while multivariate linear or nonparametric regression fail to leverage the correlation information among multivariate response variables. We propose a novel approach named nonparametric principal subspace regression to overcome these issues. By decoupling the model discrepancy, a simple and general two-step framework is …
We consider nonparametric estimation of the mean and covariance functions for functional/longitudinal data. Strong uniform convergence rates are developed for estimators that are local-linear smoothers. Our results are obtained in a unified framework in which the number of observations within each curve/cluster can be of any rate relative to the sample size. We show that the convergence rates for the procedures depend on both the number of sample curves and the number of observations on each curve. For sparse functional data, these rates are equivalent to the optimal rates in nonparametric reg…
Principal Moment Analysis is a method designed for dimension reduction, analysis and visualization of high dimensional multivariate data. It generalizes Principal Component Analysis and allows for significant statistical modeling flexibility, when approximating an unknown underlying probability distribution, by enabling direct analysis of general approximate measures. Through https://principalmomentanalysis.github.io/ we provide an implementation, together with a graphical user interface, of a simplex based version of Principal Moment Analysis.
We study least squares linear regression over $N$ uncorrelated Gaussian features that are selected in order of decreasing variance. When the number of selected features $p$ is at most the sample size $n$, the estimator under consideration coincides with the principal component regression estimator; when $p>n$, the estimator is the least $\ell_2$ norm solution over the selected features. We give an average-case analysis of the out-of-sample prediction error as $p,n,N \to \infty$ with $p/N \to α$ and $n/N \to β$, for some constants $α\in [0,1]$ and $β\in (0,1)$. In this average-case setting, the…
We analyse the properties of the Principal Fitted Components (PFC) algorithm proposed by Cook. We derive theoretical properties of the resulting estimators, including sufficient conditions under which they are $\sqrt{n}$-consistent, and explain some of the simulation results given in Cook's paper. We use techniques from random matrix theory and perturbation theory. We argue that, under Cook's model at least, the PFC algorithm should outperform the Principal Components algorithm.
The paper is concerned with asymptotic properties of the principal components analysis of functional data. The currently available results assume the existence of the fourth moment. We develop analogous results in a setting which does not require this assumption. Instead, we assume that the observed functions are regularly varying. We derive the asymptotic distribution of the sample covariance operator and of the sample functional principal components. We obtain a number of results on the convergence of moments and almost sure convergence. We apply the new theory to establish the consistency o…
This is a tutorial and survey paper on various methods for Sufficient Dimension Reduction (SDR). We cover these methods with both statistical high-dimensional regression perspective and machine learning approach for dimensionality reduction. We start with introducing inverse regression methods including Sliced Inverse Regression (SIR), Sliced Average Variance Estimation (SAVE), contour regression, directional regression, Principal Fitted Components (PFC), Likelihood Acquired Direction (LAD), and graphical regression. Then, we introduce forward regression methods including Principal Hessian Dir…
We consider the high order moments estimator of the frontier of a random pair introduced by Girard, S., Guillou, A., Stupfler, G. (2012). {\it Frontier estimation with kernel regression on high order moments}. In the present paper, we show that this estimator is strongly uniformly consistent on compact sets and its rate of convergence is given when the conditional cumulative distribution function belongs to the Hall class of distribution functions.
If X and Y are real valued random variables such that the first moments of X, Y, and XY exist and the conditional expectation of Y given X is an affine function of X, then the intercept and slope of the conditional expectation equal the intercept and slope of the least squares linear regression function, even though Y may not have a finite second moment. As a consequence, the affine in X form of the conditional expectation and zero covariance imply mean independence.
We consider finite mixtures of generalized linear models with binary output. We prove that cross moment (between the output and the regression variables) until order 3 are sufficient to identify all parameters of the model. We propose a least-squares estimation method based on those moments and we prove the consistency and the Gaussian asymptotic behavior of the estimator. We provide simulation results and comparisons with likelihood methods. Numerical experiments were conducted using the R-package Morpheus that we developed for our least-squares moment method and with the R-package flexmix fo…
In this article, we consider an imputation method to handle missing response values based on semiparametric quantile regression estimation. In the proposed method, the missing response values are generated using the estimated conditional quantile regression function at given values of covariates. We adopt the generalized method of moments for estimation of parameters defined through a general estimation equation. We demonstrate that the proposed estimator, which combines both semiparametric quantile regression imputation and generalized method of moments, has competitive edge against some of t…
This paper provides some extended results on estimating parameter matrix of several regression models when the covariate or response possesses weaker moment condition. We study the $M$-estimator of Fan et al. (Ann Stat 49(3):1239--1266, 2021) for matrix completion model with $(1+ε)$-th moment noise. The corresponding phase transition phenomenon is observed. When $1> ε>0$, the robust estimator possesses a slower convergence rate compared with previous literature. For high dimensional multiple index coefficient model, we propose an improved estimator via applying the element-wise truncation meth…
We discuss nonparametric tests for parametric specifications of regression quantiles. The test is based on the comparison of parametric and nonparametric fits of these quantiles. The nonparametric fit is a Nadaraya-Watson quantile smoothing estimator. An asymptotic treatment of the test statistic requires the development of new mathematical arguments. An approach that makes only use of plugging in a Bahadur expansion of the nonparametric estimator is not satisfactory. It requires too strong conditions on the dimension and the choice of the bandwidth. Our alternative mathematical approach requi…
Suppose the nonparametric regression function of a response variable $Y$ on covariates $X$ and $Z$ is an affine function of $X$ such that the slope $β$ and the intercept $α$ are real valued measurable functions on the range of the completely arbitrary random element $Z$. Assume that $X$ has a finite moment of order greater than or equal to $2$, $Y$ has a finite moment of conjugate order, and $α\left(Z\right)$ and $α\left(Z\right)X$ have finite first moments. Then, the nonparametric regression function equals the least squares linear regression function of $Y$ on $X$ with all the moments that a…
In this paper, we introduce principal asymmetric least squares (PALS) as a unified framework for linear and nonlinear sufficient dimension reduction. Classical methods such as sliced inverse regression (Li, 1991) and principal support vector machines (Li, Artemiou and Li, 2011) may not perform well in the presence of heteroscedasticity, while our proposal addresses this limitation by synthesizing different expectile levels. Through extensive numerical studies, we demonstrate the superior performance of PALS in terms of both computation time and estimation accuracy. For the asymptotic analysis …
We study regression models for the situation where both dependent and independent variables are square-integrable stochastic processes. Questions concerning the definition and existence of the corresponding functional linear regression models and some basic properties are explored for this situation. We derive a representation of the regression parameter function in terms of the canonical components of the processes involved. This representation establishes a connection between functional regression and functional canonical analysis and suggests alternative approaches for the implementation of…
In this paper, we study an additive model where the response variable is Hilbert-space-valued and predictors are multivariate Euclidean, and both are possibly imperfectly observed. Considering Hilbert-space-valued responses allows to cover Euclidean, compositional, functional and density-valued variables. By treating imperfect responses, we can cover functional variables taking values in a Riemannian manifold and the case where only a random sample from a density-valued response is available. This treatment can also be applied in semiparametric regression. Dealing with imperfect predictors all…
Ratios of random variables often appear in probability and statistical applications. We aim to approximate the moments of such ratios under several dependence assumptions. Extending the ideas in Collomb [C. R. Acad. Sci. Paris 285 (1977) 289--292], we propose sharper bounds for the moments of randomly weighted sums and for the $L^p$-deviations from the asymptotic normal law when the central limit theorem holds. We indicate suitable applications in finance and censored data analysis and focus on the applications in the field of functional estimation.
We propose a new method for dimension reduction in regression using the first two inverse moments. We develop corresponding weighted chi-squared tests for the dimension of the regression. The proposed method considers linear combinations of Sliced Inverse Regression (SIR) and the method using a new candidate matrix which is designed to recover the entire inverse second moment subspace. The optimal combination may be selected based on the p-values derived from the dimension tests. Theoretically, the proposed method, as well as Sliced Average Variance Estimate (SAVE), are more capable of recover…
Finite mixtures of regression models provide a flexible modeling framework for many phenomena. Using moment-based estimation of the regression parameters, we develop unbiased estimators with a minimum of assumptions on the mixture components. In particular, only the average regression model for one of the components in the mixture model is needed and no requirements on the distributions. The consistency and asymptotic distribution of the estimators is derived and the proposed method is validated through a series of simulation studies and is shown to be highly accurate. We illustrate the use of…
In functional linear regression, the parameters estimation involves solving a non necessarily well-posed problem and it has points of contact with a range of methodologies, including statistical smoothing, deconvolution and projection on finite-dimensional subspaces. We discuss the standard approach based explicitly on functional principal components analysis, nevertheless the choice of the number of basis components remains something subjective and not always properly discussed and justified. In this work we discuss inferential properties of least square estimation in this context with differ…
