This paper discusses a nonparametric regression model that naturally generalizes neural network models. The model is based on a finite number of one-dimensional transformations and can be estimated with a one-dimensional rate of convergence. The model contains the generalized additive model with unknown link function as a special case. For this case, it is shown that the additive components and link function can be estimated with the optimal rate by a smoothing spline that is the solution of a penalized least squares criterion.
It is proved that nonparametric autoregression is asymptotically equivalent in the sense of Le Cam's deficiency distance to nonparametric regression with random design as well as with regular nonrandom design.
An adaptive nonparametric estimation procedure is constructed for heteroscedastic regression when the noise variance depends on the unknown regression. A non-asymptotic upper bound for a quadratic risk (oracle inequality) is obtained
An adaptive nonparametric estimation procedure is constructed for the estimation problem of heteroscedastic regression when the noise variance depends on the unknown regression. A non-asymptotic upper bound for a quadratic risk (an oracle inequality) is constructed.
This manuscript is a supplemental document providing the omitted material for our paper entitled "Nonparametric kernel estimation of the probability density function of regression errors using estimated residuals" [arXiv:1010.0439]. The paper is submitted to Journal of Nonparametric Statistics.
This note shows that adding monotonicity or convexity constraints on the regression function does not restore well-posedness in nonparametric instrumental variable regression. The minimum distance problem without regularisation is still locally ill-posed.
In this paper, we develop nonparametric inference on spatial regression models as an extension of Lu and Tj\ostheim(2014), which develops nonparametric inference on density functions of stationary spatial processes under domain expanding and infill (DEI) asymptotics. In particular, we derive multivariate central limit theorems of mean and variance functions of nonparametric spatial regression models. Built upon those results, we propose a method to construct confidence bands for mean and variance functions.
We construct an efficient estimator for the error distribution function of the nonparametric regression model Y = r(Z) + e. Our estimator is a kernel smoothed empirical distribution function based on residuals from an under-smoothed local quadratic smoother for the regression function.
This paper concerns the estimation of the regression function at a given point in nonparametric heteroscedastic models with Gaussian noise or with noise having unknown distribution. In the two cases an asymptotically efficient kernel estimator is constructed for the minimax absolute error risk.
In this paper, we are concerned with nonparametric estimation of the multivariate regression function in the presence of right censored data. More precisely, we propose a statistic that is shown to be asymptotically normally distributed under the additive assumption, and that could be used to test for additivity in the censored regression setting.
We establish the asymptotic normality of the kernel type estimator for the regression function constructed from quasi-associated data when the explanatory variable takes its values in a separable Hilbert space.
This paper is devoted to the nonparametric estimation of the derivative of the regression function in a nonparametric regression model. We implement a very efficient and easy to handle statistical procedure based on the derivative of the recursive Nadaraya-Watson estimator. We establish the almost sure convergence as well as the asymptotic normality for our estimates. We also illustrate our nonparametric estimation procedure on simulated and real life data associated with sea shores water quality and valvometry.
The detection of influential observations for the standard least squares regression model is a question that has been extensively studied. LAD regression diagnostics offers alternative approaches whose main feature is the robustness. In this paper a new approach for nonparametric detection of influencial observations in LAD regression models is presented and compared with other classical methods of diagnostics.
In this study, we develop an asymptotic theory of nonparametric regression for a locally stationary functional time series. First, we introduce the notion of a locally stationary functional time series (LSFTS) that takes values in a semi-metric space. Then, we propose a nonparametric model for LSFTS with a regression function that changes smoothly over time. We establish the uniform convergence rates of a class of kernel estimators, the Nadaraya-Watson (NW) estimator of the regression function, and a central limit theorem of the NW estimator.
We introduce and study a local linear nonparametric regression estimator for censorship model. The main goal of this paper is, to establish the uniform almost sure consistency result with rate over a compact set for the new estimate. To support our theoretical result, a simulation study has been done to make comparison with the classical regression estimator.
This paper discusses a design-dependent nature of variance in nonparametric link regression aiming at predicting a mean outcome at a link, i.e., a pair of nodes, based on currently observed data comprising covariates at nodes and outcomes at links.
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper (2007) for estimation of unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk. It means that the asymptotic quadratic risk for this procedure coincides with a sharp lower bound.
We analyze the statistical properties of nonparametric regression estimators using covariates which are not directly observable, but have be estimated from data in a preliminary step. These so-called generated covariates appear in numerous applications, including two-stage nonparametric regression, estimation of simultaneous equation models or censored regression models. Yet so far there seems to be no general theory for their impact on the final estimator's statistical properties. Our paper provides such results. We derive a stochastic expansion that characterizes the influence of the generat…
We show that nonparametric regression is asymptotically equivalent in Le Cam's sense with a sequence of Gaussian white noise experiments as the number of observations tends to infinity. We propose a general constructive framework based on approximation spaces, which permits to achieve asymptotic equivalence even in the cases of multivariate and random design.
In this paper we prove large deviations principles for the Nadaraya-Watson estimator of the regression of a real-valued variable with a functional covariate. Under suitable conditions, we show pointwise and uniform large deviations theorems with good rate functions.
We discuss nonparametric tests for parametric specifications of regression quantiles. The test is based on the comparison of parametric and nonparametric fits of these quantiles. The nonparametric fit is a Nadaraya-Watson quantile smoothing estimator. An asymptotic treatment of the test statistic requires the development of new mathematical arguments. An approach that makes only use of plugging in a Bahadur expansion of the nonparametric estimator is not satisfactory. It requires too strong conditions on the dimension and the choice of the bandwidth. Our alternative mathematical approach requi…
A nonparametric regression setting is considered with a real-valued covariate and responses from a metric space. One may approach this setting via Fréchet regression, where the value of the regression function at each point is estimated via a Fréchet mean calculated from an estimated objective function. A second approach is geodesic regression, which builds upon fitting geodesics to observations by a least squares method. These approaches are applied to transform two of the most important nonparametric regression estimators in statistics to the metric setting -- the local linear regression est…
Suppose the nonparametric regression function of a response variable $Y$ on covariates $X$ and $Z$ is an affine function of $X$ such that the slope $β$ and the intercept $α$ are real valued measurable functions on the range of the completely arbitrary random element $Z$. Assume that $X$ has a finite moment of order greater than or equal to $2$, $Y$ has a finite moment of conjugate order, and $α\left(Z\right)$ and $α\left(Z\right)X$ have finite first moments. Then, the nonparametric regression function equals the least squares linear regression function of $Y$ on $X$ with all the moments that a…
In this note, we consider the problem of existence of adaptive confidence bands in the fixed design regression model, adapting ideas in Hoffmann and Nickl (2011) to the present case. In the course of the proof, we show that sup-norm adaptive estimators exist as well in regression.
In the context of nonparametric regression, we study conditions under which the consistency (and rates of convergence) of estimators built from discretely sampled curves can be derived from the consistency of estimators based on the unobserved whole trajectories. As a consequence, we derive asymptotic results for most of the regularization techniques used in functional data analysis, including smoothing and basis representation.
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper, 2007, for estimating a unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk, i.e. the asymptotic quadratic risk for this procedure coincides with the Pinsker constant which gives a sharp lower bound for the quadratic risk over all possible estimators.
In this paper, we built a new nonparametric regression estimator with the local linear method by using the mean squared relative error as a loss function when the data are subject to random right censoring. We establish the uniform almost sure consistency with rate over a compact set of the proposed estimator. Some simulations are given to show the asymptotic behavior of the estimate in different cases.
This paper deals with the nonparametric density estimation of the regression error term assuming its independence with the covariate. The difference between the feasible estimator which uses the estimated residuals and the unfeasible one using the true residuals is studied. An optimal choice of the bandwidth used to estimate the residuals is given. We also study the asymptotic normality of the feasible kernel estimator and its rate-optimality.
This paper develops a method to construct uniform confidence bands for a nonparametric regression function where a predictor variable is subject to a measurement error. We allow for the distribution of the measurement error to be unknown, but assume the availability of validation data or repeated measurements on the latent predictor variable. The proposed confidence band builds on the deconvolution kernel estimation and a novel application of the multiplier bootstrap method. We establish asymptotic validity of the proposed confidence band. To our knowledge, this is the first paper to derive as…
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper, 2007, for estimating an unknown nonparametric regression. %\cite{GaPe1}. We prove that this procedure is asymptotically efficient for a quadratic risk, i.e. the asymptotic quadratic risk for this procedure coincides with the Pinsker constant which gives a sharp lower bound for the quadratic risk over all possible estimates
We develop a unifying framework for Bayesian nonparametric regression to study the rates of contraction with respect to the integrated $L_2$-distance without assuming the regression function space to be uniformly bounded. The framework is very flexible and can be applied to a wide class of nonparametric prior models. Three non-trivial applications of the proposed framework are provided: The finite random series regression of an $α$-Hölder function, with adaptive rates of contraction up to a logarithmic factor; The un-modified block prior regression of an $α$-Sobolev function, with adaptive-and…
We consider the nonparametric robust estimation problem for regression models in continuous time with semi-Markov noises. An adaptive model selection procedure is proposed. Under general moment conditions on the noise distribution a sharp non-asymptotic oracle inequality for the robust risks is obtained and the robust efficiency is shown. It turns out that for semi-Markov models the robust minimax convergence rate may be faster or slower than the classical one.
The main purpose is to estimate the regression function of a real random variable with functional explanatory variable by using a recursive nonparametric kernel approach. The mean square error and the almost sure convergence of a family of recursive kernel estimates of the regression function are derived. These results are established with rates and precise evaluation of the constant terms. Also, a central limit theorem for this class of estimators is established. The method is evaluated on simulations and real data set studies.
In this work, we develop and study an empirical projection operator scheme for solving nonparametric regression problems. This scheme is based on an approximate projection of the regression function over a suitable reproducing kernel Hilbert space (RKHS). The RKHS considered in this paper are generated by the Mercer kernels given by the Legendre Christoffel-Darboux and convolution Sinc kernels. We provide error and convergence analysis of the proposed scheme under the assumption that the regression function belongs to some suitable functional spaces. We also consider the popular RKHS regulariz…
Most results in nonparametric regression theory are developed only for the case of additive noise. In such a setting many smoothing techniques including wavelet thresholding methods have been developed and shown to be highly adaptive. In this paper we consider nonparametric regression in exponential families with the main focus on the natural exponential families with a quadratic variance function, which include, for example, Poisson regression, binomial regression and gamma regression. We propose a unified approach of using a mean-matching variance stabilizing transformation to turn the relat…
Principled nonparametric tests for regression curvature in $\mathbb{R}^{d}$ are often statistically and computationally challenging. This paper introduces the stratified incomplete local simplex (SILS) tests for joint concavity of nonparametric multiple regression. The SILS tests with suitable bootstrap calibration are shown to achieve simultaneous guarantees on dimension-free computational complexity, polynomial decay of the uniform error-in-size, and power consistency for general (global and local) alternatives. To establish these results, a general theory for incomplete $U$-processes with s…
In this paper, a lower bound is determined in the minimax sense for change point estimators of the first derivative of a regression function in the fractional white noise model. Similar minimax results presented previously in the area focus on change points in the derivatives of a regression function in the white noise model or consider estimation of the regression function in the presence of correlated errors.
We consider the recursive estimation of a regression functional where the explanatory variables take values in some functional space. We prove the almost sure convergence of such estimates for dependent functional data. Also we derive the mean quadratic error of the considered class of estimators. Our results are established with rates and asymptotic appear bounds, under strong mixing condition.
We consider nonparametric estimation of mean regression and conditional variance (or volatility) functions in nonlinear stochastic regression models. Simultaneous confidence bands are constructed and the coverage probabilities are shown to be asymptotically correct. The imposed dependence structure allows applications in many linear and nonlinear auto-regressive processes. The results are applied to the S&P 500 Index data.
Asymptotic equivalence in Le Cam's sense for nonparametric regression experiments is extended to the case of non-regular error densities, which have jump discontinuities at their endpoints. We prove asymptotic equivalence of such regression models and the observation of two independent Poisson point processes which contain the target curve as the support boundary of its intensity function. The intensity of the point processes is of order of the sample size $n$ and involves the jump sizes as well as the design density. The statistical model significantly differs from regression problems with Ga…
This paper considers a general class of nonparametric time series regression models where the regression function can be time-dependent. We establish an asymptotic theory for estimates of the time-varying regression functions. For this general class of models, an important issue in practice is to address the necessity of modeling the regression function as nonlinear and time-varying. To tackle this, we propose an information criterion and prove its selection consistency property. The results are applied to the U.S. Treasury interest rate data.
This paper deals with the consistency of the least squares estimator of a convex regression function when the predictor is multidimensional. We characterize and discuss the computation of such an estimator via the solution of certain quadratic and linear programs. Mild sufficient conditions for the consistency of this estimator and its subdifferentials in fixed and stochastic design regression settings are provided. We also consider a regression function which is known to be convex and componentwise nonincreasing and discuss the characterization, computation and consistency of its least square…
In this paper we consider the nonparametric estimation of density and regression functions with non-negative support using a gamma kernel procedure introduced by Chen (2000). Strong uniform consistency and asymptotic normality of the corresponding estimators are established under a general ergodic assumption on the data generation process. Our results generalize those of Shi and Song (2016), obtained in the classic i.i.d. framework, and the works of Bouezmarni and Rombouts (2008, 2010b) and Gospodinov and Hirukawa (2007) for mixing time series.
Efficient estimation under bias sampling, censoring or truncation is a difficult question which has been partially answered and the usual estimators are not always consistent. Several biased designs are considered for models with variables $(X,Y)$ where $Y$ is an indicator and $X$ an explanatory variable, or for continuous variables $(X,Y)$. The identifiability of the models are discussed. New nonparametric estimators of the regression functions and conditional quantiles are proposed.
Asymptotic equivalence results for nonparametric regression experiments have always assumed that the variances of the observations are known. In practice, however the variance of each observation is generally considered to be an unknown nuisance parameter. We establish an asymptotic approximation to the nonparametric regression experiment when the value of the variance is an additional parameter to be estimated or tested. This asymptotically equivalent experiment has two components: the first contains all the information about the variance and the second has all the information about the mean.…
We consider the problem of recovering of continuous multi-dimensional functions from the noisy observations over the regular grid. Our focus is at the adaptive estimation in the case when the function can be well recovered using a linear filter, which can depend on the unknown function itself. In the companion paper "Nonparametric Denoising of Signals with Unknown Local Structure, I: Oracle Inequalities" we have shown in the case when there exists an adapted time-invariant filter, which locally recovers "well" the unknown signal, there is a numerically efficient construction of an adaptive fil…
In this paper we propose a new test for the hypothesis of a constant coefficient of variation in the common nonparametric regression model. The test is based on an estimate of the $L^2$-distance between the square of the regression function and variance function. We prove asymptotic normality of a standardized estimate of this distance under the null hypothesis and fixed alternatives and the finite sample properties of a corresponding bootstrap test are investigated by means of a simulation study. The results are applicable to stationary processes with the common mixing conditions and are used…
We propose a new, two-step empirical Bayes-type of approach for neural networks. We show in context of the nonparametric regression model that the procedure (up to a logarithmic factor) provides optimal recovery of the underlying functional parameter of interest and provides Bayesian credible sets with frequentist coverage guarantees. The approach requires fitting the neural network only once, hence it is substantially faster than Bootstrapping type approaches. We demonstrate the applicability of our method over synthetic data, observing good estimation properties and reliable uncertainty quan…
We construct nonparametric confidence sets for regression functions using wavelets that are uniform over Besov balls. We consider both thresholding and modulation estimators for the wavelet coefficients. The confidence set is obtained by showing that a pivot process, constructed from the loss function, converges uniformly to a mean zero Gaussian process. Inverting this pivot yields a confidence set for the wavelet coefficients, and from this we obtain confidence sets on functionals of the regression curve.
In this paper, we consider the nonparametric regression problem with multivariate predictors. We provide a characterization of the degrees of freedom and divergence for estimators of the unknown regression function, which are obtained as outputs of linearly constrained quadratic optimization procedures, namely, minimizers of the least squares criterion with linear constraints and/or quadratic penalties. As special cases of our results, we derive explicit expressions for the degrees of freedom in many nonparametric regression problems, e.g., bounded isotonic regression, multivariate (penalized)…
In several different fields, there is interest in analyzing the upper or lower tail quantile of the underlying distribution rather than mean or center quantile. However, the investigation of the tail quantile is difficult because of data sparsity. In this paper, we attempt to develop nonparametric quantile regression for the extremal quantile level. In extremal quantile regression, there are two types of technical conditions of the order of convergence of the quantile level: intermediate order or extreme order. For the intermediate order quantile, the ordinary nonparametric estimator is used. …
We consider the problem of estimating the value of a linear functional in nonparametric instrumental regression, where in the presence of an instrument W a response Y is modeled in dependence of an endogenous explanatory variable Z. The proposed estimator is based on dimension reduction and additional thresholding. The minimax optimal rate of convergence of the estimator is derived assuming that the structural function and the representer of the linear functional belong to some ellipsoids which are in a certain sense linked to the conditional expectation operator of Z given W. We illustrate th…
In this paper we consider a regression model that allows for time series covariates as well as heteroscedasticity with a regression function that is modelled nonparametrically. We assume that the regression function changes at some unknown time $\lfloor ns_0\rfloor$, $s_0\in(0,1)$, and our aim is to estimate the (rescaled) change point $s_0$. The considered estimator is based on a Kolmogorov-Smirnov functional of the marked empirical process of residuals. We show consistency of the estimator and prove a rate of convergence of $O_P(n^{-1})$ which in this case is clearly optimal as there are onl…
We consider testing the significance of a subset of covariates in a nonparametric regression. These covariates can be continuous and/or discrete. We propose a new kernel-based test that smoothes only over the covariates appearing under the null hypothesis, so that the curse of dimensionality is mitigated. The test statistic is asymptotically pivotal and the rate of which the test detects local alternatives depends only on the dimension of the covariates under the null hypothesis. We show the validity of wild bootstrap for the test. In small samples, our test is competitive compared to existing…
We present a greedy method for simultaneously performing local bandwidth selection and variable selection in nonparametric regression. The method starts with a local linear estimator with large bandwidths, and incrementally decreases the bandwidth of variables for which the gradient of the estimator with respect to bandwidth is large. The method--called rodeo (regularization of derivative expectation operator)--conducts a sequence of hypothesis tests to threshold derivatives, and is easy to implement. Under certain assumptions on the regression function and sampling density, it is shown that t…
In this paper we develop the James - Stein improved estimation method for a nonparametric periodic function observed with the Levy noises in continuous time. An adaptive model selection procedure based on the improved weighted least square estimates is constructed. The improvement effect for the nonparametric models is obtained. It turns out that in the nonasymptotic studies the accuracy improvement for nonparametric problems is more significantly than for the parametric one. Moreover, sharp oracle inequalities for the robust risks have been shown and the efficiency property for the improved m…
