In this article, we study the convergence behavior of the regularization-based algorithm for solving the polynomial regression model when both input data and responses are from infinite-dimensional Hilbert spaces. We derive convergence rates for estimation and prediction error by employing general (spectral) regularization under a general smoothness condition without imposing any additional conditions on the index function. We also establish lower bounds for any learning algorithm to explain the optimality of our convergence rates.
In a seminal paper \cite{studden1968} characterized $c$-optimal designs in regression models, where the regression functions form a Chebyshev system. He used these results to determine the optimal design for estimating the individual coefficients in a polynomial regression model on the interval $[-1,1]$ explicitly. In this note we identify the optimal design for estimating the individual coefficients in a polynomial regression model with no intercept (here the regression functions do not form a Chebyshev system).
In this work, we investigate Gaussian process regression used to recover a function based on noisy observations. We derive upper and lower error bounds for Gaussian process regression with possibly misspecified correlation functions. The optimal convergence rate can be attained even if the smoothness of the imposed correlation function exceeds that of the true correlation function and the sampling scheme is quasi-uniform. As byproducts, we also obtain convergence rates of kernel ridge regression with misspecified kernel function, where the underlying truth is a deterministic function. The conv…
In statistics, experimental designs are methods for making efficient experiments. E-optimal designs are the multisets of experimental conditions which minimize the maximum axis of the confidence ellipsoid of estimators. The aim of this thesis is to propose a new algorithm for constructing E-optimal designs approximately for weighted polynomial regression with a nonnegative weight function. First, an algorithm to calculate E-optimal designs for weighted polynomial regression of particular weight functions is discussed. Next a new algorithm for constructing E-optimal designs approximately is pro…
In this note we consider the optimal design problem for estimating the slope of a polynomial regression with no intercept at a given point, say z. In contrast to previous work, which considers symmetric design spaces we investigate the model on the interval $[0, a]$ and characterize those values of $z$, where an explicit solution of the optimal design is possible.
We reveal the phenomenon that ``naive'' multivariate local polynomial regression can adapt to local smooth lower dimensional structure in the sense that it achieves the optimal convergence rate for nonparametric estimation of regression functions belonging to a Sobolev space when the predictor variables live on or close to a lower dimensional manifold.
We propose a kernel mixture of polynomials prior for Bayesian nonparametric regression. The regression function is modeled by local averages of polynomials with kernel mixture weights. We obtain the minimax-optimal rate of contraction of the full posterior distribution up to a logarithmic factor that adapts to the smoothness level of the true function by estimating metric entropies of certain function classes. We also provide a frequentist sieve maximum likelihood estimator with a near-optimal convergence rate. We further investigate the application of the kernel mixture of polynomials to the …
We consider the problem of isotonic regression, where the underlying signal $x$ is assumed to satisfy a monotonicity constraint, that is, $x$ lies in the cone $\{ x\in\mathbb{R}^n : x_1 \leq \dots \leq x_n\}$. We study the isotonic projection operator (projection to this cone), and find a necessary and sufficient condition characterizing all norms with respect to which this projection is contractive. This enables a simple and non-asymptotic analysis of the convergence properties of isotonic regression, yielding uniform confidence bands that adapt to the local Lipschitz properties of the signal…
Improvements in technology lead to increasing availability of large data sets which makes the need for data reduction and informative subsamples ever more important. In this paper we construct $ D $-optimal subsampling designs for polynomial regression in one covariate for invariant distributions of the covariate. We study quadratic regression more closely for specific distributions. In particular we make statements on the shape of the resulting optimal subsampling designs and the effect of the subsample size on the design. To illustrate the advantage of the optimal subsampling designs we exam…
We consider heteroscedastic nonparametric regression models, when both the mean function and variance function are unknown and to be estimated with nonparametric approaches. We derive convergence rates of posterior distributions for this model with different priors, including splines and Gaussian process priors. The results are based on the general ones on the rates of convergence of posterior distributions for independent, non-identically distributed observations, and are established for both of the cases with random covariates, and deterministic covariates. We also illustrate that the result…
Shape restricted regressions, including isotonic regression and concave regression as special cases, are studied using priors on Bernstein polynomials and Markov chain Monte Carlo methods. These priors have large supports, select only smooth functions, can easily incorporate geometric information into the prior, and can be generated without computational difficulty. Algorithms generating priors and posteriors are proposed, and simulation studies are conducted to illustrate the performance of this approach. Comparisons with the density-regression method of Dette et al. (2006) are included.
A polynomial that is nonnegative over a given interval is called a positive polynomial. The set of such positive polynomials forms a closed convex cone $K$. In this paper, we consider the likelihood ratio test for the hypothesis of positivity that the estimand polynomial regression curve is a positive polynomial. By considering hierarchical hypotheses including the hypothesis of positivity, we define nested likelihood ratio tests, and derive their null distributions as mixtures of chi-square distributions by using the volume-of-tubes method. The mixing probabilities are obtained by utilizing t…
We consider a nonlinear polynomial regression model in which we wish to test the null hypothesis of structural stability in the regression parameters against the alternative of a break at an unknown time. We derive the extreme value distribution of a maximum-type test statistic which is asymptotically equivalent to the maximally selected likelihood ratio. The resulting test is easy to apply and has good size and power, even in small samples.
In this work, we construct a stable and fairly fast estimator for solving non-parametric multidimensional regression problems. The proposed estimator is based on the use of multivariate Jacobi polynomials that generate a basis for a reduced size of $d-$variate finite dimensional polynomial space. An ANOVA decomposition trick has been used for building this later polynomial space. Also, by using some results from the theory of positive definite random matrices, we show that the proposed estimator is stable under the condition that the i.i.d. random sampling points for the different covariates o…
Recent results in nonparametric regression show that for deep learning, i.e., for neural network estimates with many hidden layers, we are able to achieve good rates of convergence even in case of high-dimensional predictor variables, provided suitable assumptions on the structure of the regression function are imposed. The estimates are defined by minimizing the empirical $L_2$ risk over a class of neural networks. In practice it is not clear how this can be done exactly. In this article we introduce a new neural network regression estimate where most of the weights are chosen regardless of t…
Nonparametric density and regression estimators commonly depend on a bandwidth. The asymptotic properties of these estimators have been widely studied when bandwidths are nonstochastic. In practice, however, in order to improve finite sample performance of these estimators, bandwidths are selected by data driven methods, such as cross-validation or plug-in procedures. As a result nonparametric estimators are usually constructed using stochastic bandwidths. In this paper we establish the asymptotic equivalence in probability of local polynomial regression estimators under stochastic and nonstoc…
Kernel ridge regression is an important nonparametric method for estimating smooth functions. We introduce a new set of conditions, under which the actual rates of convergence of the kernel ridge regression estimator under both the L_2 norm and the norm of the reproducing kernel Hilbert space exceed the standard minimax rates. An application of this theory leads to a new understanding of the Kennedy-O'Hagan approach for calibrating model parameters of computer simulation. We prove that, under certain conditions, the Kennedy-O'Hagan calibration estimator with a known covariance function converg…
We consider nonparametric functional regression when both predictors and responses are functions. More specifically, we let $(X_1,Y_1),...,(X_n,Y_n)$ be random elements in $\mathcal{F}\times\mathcal{H}$ where $\mathcal{F}$ is a semi-metric space and $\mathcal{H}$ is a separable Hilbert space. Based on a recently introduced notion of weak dependence for functional data, we showed the almost sure convergence rates of both the Nadaraya-Watson estimator and the nearest neighbor estimator, in a unified manner. Several factors, including functional nature of the responses, the assumptions on the fun…
We generalize a method for proving uniform in bandwidth consistency results for kernel type estimators developed by the two last named authors. Such results are shown to be useful in establishing consistency of local polynomial estimators of the regression function.
We present a new approach to the design of D-optimal experiments with multivariate polynomial regressions on compact semi-algebraic design spaces. We apply the moment-sum-of-squares hierarchy of semidefinite programming problems to solve numerically and approximately the optimal design problem. The geometry of the design is recovered with semidefinite programming duality theory and the Christoffel polynomial.
Under a standard assumption in complexity theory (NP not in P/poly), we demonstrate a gap between the minimax prediction risk for sparse linear regression that can be achieved by polynomial-time algorithms, and that achieved by optimal algorithms. In particular, when the design matrix is ill-conditioned, the minimax prediction loss achievable by polynomial-time algorithms can be substantially greater than that of an optimal algorithm. This result is the first known gap between polynomial and optimal algorithms for sparse linear regression, and does not depend on conjectures in average-case com…
In this paper we develop the theory of parametric polynomial regression in Riemannian manifolds and Lie groups. We show application of Riemannian polynomial regression to shape analysis in Kendall shape space. Results are presented, showing the power of polynomial regression on the classic rat skull growth data of Bookstein as well as the analysis of the shape changes associated with aging of the corpus callosum from the OASIS Alzheimer's study.
This paper presents uniform convergence rates for kernel regression estimators, in the setting of a structural nonlinear cointegrating regression model. We generalise the existing literature in three ways. First, the domain to which these rates apply is much wider than has been previously considered, and can be chosen so as to contain as large a fraction of the sample as desired in the limit. Second, our results allow the regression disturbance to be serially correlated, and cross-correlated with the regressor; previous work on this problem (of obtaining uniform rates) having been confined ent…
We use local polynomial fitting to estimate the nonparametric M-regression function for strongly mixing stationary processes $\{(Y_{i},\underline{X}_{i})\}$. We establish a strong uniform consistency rate for the Bahadur representation of estimators of the regression function and its derivatives. These results are fundamental for statistical inference and for applications that involve plugging in such estimators into other functionals where some control over higher order terms are required. We apply our results to the estimation of an additive M-regression model.
Suppose that $n$ statistical units are observed, each following the model $Y(x_j)=m(x_j)+ ε(x_j),\, j=1,...,N,$ where $m$ is a regression function, $0 \leq x_1 <...<x_N \leq 1$ are observation times spaced according to a sampling density $f$, and $ε$ is a continuous-time error process having mean zero and regular covariance function. Considering the local polynomial estimation of $m$ and its derivatives, we derive asymptotic expressions for the bias and variance as $n,N\to\infty$. Such results are particularly relevant in the context of functional data where essential information is contained …
Non-parametric estimation of functions as well as their derivatives by means of local-polynomial regression is a subject that was studied in the literature since the late 1970's. Given a set of noisy samples of a $\mathcal{C}^k$ smooth function, we perform a local polynomial fit, and by taking its $m$-th derivative we obtain an estimate for the $m$-th function derivative. The known optimal rates of convergence for this problem for a $k$-times smooth function $f:\mathbb{R}^d \to \mathbb{R}$ are $n^{-\frac{k-m}{2k + d}}$. However in modern applications it is often the case that we have to estima…
We consider a nonparametric regression setup, where the covariate is a random element in a complete separable metric space, and the parameter of interest associated with the conditional distribution of the response lies in a separable Banach space. We derive the optimum convergence rate for the kernel estimate of the parameter in this setup. The small ball probability in the covariate space plays a critical role in determining the asymptotic variance of kernel estimates. Unlike the case of finite dimensional covariates, we show that the asymptotic orders of the bias and the variance of the est…
In this article, we investigate posterior convergence of nonparametric binary and Poisson regression under possible model misspecification, assuming general stochastic process prior with appropriate properties. Our model setup and objective for binary regression is similar to that of Ghosal and Roy (2006) where the authors have used the approach of entropy bound and exponentially consistent tests with the sieve method to achieve consistency with respect to their Gaussian process prior. In contrast, for both binary and Poisson regression, using general stochastic process prior, our approach inv…
We consider nonparametric estimation of the mean and covariance functions for functional/longitudinal data. Strong uniform convergence rates are developed for estimators that are local-linear smoothers. Our results are obtained in a unified framework in which the number of observations within each curve/cluster can be of any rate relative to the sample size. We show that the convergence rates for the procedures depend on both the number of sample curves and the number of observations on each curve. For sparse functional data, these rates are equivalent to the optimal rates in nonparametric reg…
Nonparametric regression with random design is considered. Estimates are defined by minimzing a penalized empirical $L_2$ risk over a suitably chosen class of neural networks with one hidden layer via gradient descent. Here, the gradient descent procedure is repeated several times with randomly chosen starting values for the weights, and from the list of constructed estimates the one with the minimal empirical $L_2$ risk is chosen. Under the assumption that the number of randomly chosen starting values and the number of steps for gradient descent are sufficiently large it is shown that the res…
For a class of martingales, this paper provides a framework on the uniform consistency with broad applicability. The main condition imposed is only related to the conditional variance of the martingale, which holds true for stationary mixing time series, stationary iterated random function, Harris recurrent Markov chains and $I(1)$ processes with innovations being a linear process. Using the established results, this paper investigates the uniform convergence of the Nadaraya-Watson estimator in a non-linear cointegrating regression model. Our results not only provide sharp convergence rate, bu…
It is shown that any linear estimator that satisfies the moment conditions up to order $p$ is equivalent to a local polynomial regression of order $p$ with some non-negative weight function if and only if the kernel has at most $p$ sign changes. If the data points are placed symmetrically about the estimation point, a linear weighting function is equivalent to the standard quadratic weighting function.
This paper deals with the speed of convergence of the learning curve in a Gaussian process regression framework. The learning curve describes the average generalization error of the Gaussian process used for the regression. More specifically, it is defined in this paper as the integral of the mean squared error over the input parameter space with respect to the probability measure of the input parameters. The main result is the proof of a theorem giving the mean squared error in function of the number of observations for a large class of kernels and for any dimension when the number of observa…
This paper provides ANOVA inference for nonparametric local polynomial regression (LPR) in analogy with ANOVA tools for the classical linear regression model. A surprisingly simple and exact local ANOVA decomposition is established, and a local R-squared quantity is defined to measure the proportion of local variation explained by fitting LPR. A global ANOVA decomposition is obtained by integrating local counterparts, and a global R-squared and a symmetric projection matrix are defined. We show that the proposed projection matrix is asymptotically idempotent and asymptotically orthogonal to it…
We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stopping. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. Following the setting introduced in earlier related literature, we study so-called "fast convergence rates" depending on the regularity of the target regression function (measured by a source condition in terms of the kernel integra…
We consider a general monotone regression estimation where we allow for independent and dependent regressors. We propose a modification of the classical isotonic least squares estimator and establish its rate of convergence for the integrated $L_1$-loss function. The methodology captures the shape of the data without assuming additivity or a parametric form for the regression function. Furthermore, the degree of smoothing is chosen automatically and no auxiliary tuning is required for the theoretical analysis. Some simulations and two real data illustrations complement the study of the propose…
We consider a nonparametric regression model $Y=r(X)+\varepsilon$ with a random covariate $X$ that is independent of the error $\varepsilon$. Then the density of the response $Y$ is a convolution of the densities of $\varepsilon$ and $r(X)$. It can therefore be estimated by a convolution of kernel estimators for these two densities, or more generally by a local von Mises statistic. If the regression function has a nowhere vanishing derivative, then the convolution estimator converges at a parametric rate. We show that the convergence holds uniformly, and that the corresponding process obeys a …
The paper continues the authors' work on the adaptive Wynn algorithm in a nonlinear regression model. In the present paper it is shown that if the mean response function satisfies a condition of `saturated identifiability', which was introduced by Pronzato \cite{Pronzato}, then the adaptive least squares estimators are strongly consistent. The condition states that the regression parameter is identifiable under any saturated design, i.e., the values of the mean response function at any $p$ distinct design points determine the parameter point uniquely where, typically, $p$ is the dimension of t…
In functional linear regression, the slope ``parameter'' is a function. Therefore, in a nonparametric context, it is determined by an infinite number of unknowns. Its estimation involves solving an ill-posed problem and has points of contact with a range of methodologies, including statistical smoothing and deconvolution. The standard approach to estimating the slope function is based explicitly on functional principal components analysis and, consequently, on spectral decomposition in terms of eigenvalues and eigenfunctions. We discuss this approach in detail and show that in certain circumst…
The use of MCMC algorithms in high dimensional Bayesian problems has become routine. This has spurred so-called convergence complexity analysis, the goal of which is to ascertain how the convergence rate of a Monte Carlo Markov chain scales with sample size, $n$, and/or number of covariates, $p$. This article provides a thorough convergence complexity analysis of Albert and Chib's (1993) data augmentation algorithm for the Bayesian probit regression model. The main tools used in this analysis are drift and minorization conditions. The usual pitfalls associated with this type of analysis are av…
We propose a method for incorporating variable selection into local polynomial regression. This can improve the accuracy of the regression by extending the bandwidth in directions corresponding to those variables judged to be are unimportant. It also increases our understanding of the dataset by highlighting areas where these variables are redundant. The approach has the potential to effect complete variable removal as well as perform partial removal when a variable redundancy applies only to particular regions of the data. We define a nonparametric oracle property and show that this is more t…
We consider nonparametric estimation of a regression function for a situation where precisely measured predictors are used to estimate the regression curve for coarsened, that is, less precise or contaminated predictors. Specifically, while one has available a sample $(W_1,Y_1),...,(W_n,Y_n)$ of independent and identically distributed data, representing observations with precisely measured predictors, where $\mathrm{E}(Y_i|W_i)=g(W_i)$, instead of the smooth regression function $g$, the target of interest is another smooth regression function $m$ that pertains to predictors $X_i$ that are nois…
In multivariate regression estimation, the rate of convergence depends on the dimension of the regressor. This fact, known as the curse of the dimensionality, motivated several works. The additive model, introduced by Stone (10), offers an efficient response to this problem. In the setting of continuous time processes, using the marginal integration method, we obtain the quadratic convergence rate and the asymptotic normality of the components of the additive model.
We look into the nonparametric regression estimation with additive and multiplicative noise and construct adaptive thresholding estimators based on Laguerre series. The proposed approach achieves asymptotically near-optimal convergence rates when the unknown function belongs to Laguerre-Sobolev space. We consider the problem under two noise structures; (1) { i.i.d.} Gaussian errors and (2) long-memory Gaussian errors. In the { i.i.d.} case, our convergence rates are similar to those found in the literature. In the long-memory case, the convergence rates depend on the long-memory parameters onl…
This paper develops a general theory on rates of convergence of penalized spline estimators for function estimation when the likelihood functional is concave in candidate functions, where the likelihood is interpreted in a broad sense that includes conditional likelihood, quasi-likelihood, and pseudo-likelihood. The theory allows all feasible combinations of the spline degree, the penalty order, and the smoothness of the unknown functions. According to this theory, the asymptotic behaviors of the penalized spline estimators depends on interplay between the spline knot number and the penalty pa…
This article contributes to the current statistical theory of deep neural networks (DNNs). It was shown that DNNs are able to circumvent the so--called curse of dimensionality in case that suitable restrictions on the structure of the regression function hold. In most of those results the tuning parameter is the sparsity of the network, which describes the number of non-zero weights in the network. This constraint seemed to be the key factor for the good rate of convergence results. Recently, the assumption was disproved. In particular, it was shown that simple fully connected DNNs can achieve…
We study the properties of nonparametric least squares regression using deep neural networks. We derive non-asymptotic upper bounds for the prediction error of the empirical risk minimizer of feedforward deep neural regression. Our error bounds achieve minimax optimal rate and significantly improve over the existing ones in the sense that they depend polynomially on the dimension of the predictor, instead of exponentially on dimension. We show that the neural regression estimator can circumvent the curse of dimensionality under the assumption that the predictor is supported on an approximate l…
A new nonparametric estimator of a convex regression function in any dimension is proposed and its convergence properties are studied. We start by using any estimator of the regression function and we \emph{convexify} it by taking the convex envelope of a sample of the approximation obtained. We prove that the uniform rate of convergence of the estimator is maintained after the convexification is applied. The finite sample properties of the new estimator are investigated by means of a simulation study and the application of the new method is demonstrated in examples.
We consider the residual empirical process in random design regression with long memory errors. We establish its limiting behaviour, showing that its rates of convergence are different from the rates of convergence for to the empirical process based on (unobserved) errors. Also, we study a residual empirical process with estimated parameters. Its asymptotic distribution can be used to construct Kolmogorov-Smirnov, Cramér-Smirnov-von Mises, or other goodness-of-fit tests. Theoretical results are justified by simulation studies.
n this paper we consider polynomial cointegrating relationships among stationary processes with long range dependence. We express the regression functions in terms of Hermite polynomials and we consider a form of spectral regression around frequency zero. For these estimates, we establish consistency by means of a more general result on continuously averaged estimates of the spectral density matrix at frequency zero
