This paper presents a Bayesian sampling approach to bandwidth estimation for the local linear estimator of the regression function in a nonparametric regression model. In the Bayesian sampling approach, the error density is approximated by a location-mixture density of Gaussian densities with means the individual errors and variance a constant parameter. This mixture density has the form of a kernel density estimator of errors and is referred to as the kernel-form error density (c.f., Zhang et al., 2014). While Zhang et al. (2014) use the local constant (also known as the Nadaraya- Watson) estimator to estimate the regression function, we extend this to the local linear estimator, which produces more accurate estimation. The proposed investigation is motivated by the lack of data-driven methods for simultaneously choosing bandwidths in the local linear estimator of the regression function and kernel-form error density. Treating bandwidths as parameters, we derive an approximate (pseudo) likelihood and a posterior. A simulation study shows that the proposed bandwidth estimation outperforms the rule-of-thumb and cross-validation methods under the criterion of integrated squared errors. The proposed bandwidth estimation method is validated through a nonparametric regression model involving firm ownership concentration, and a model involving state-price density estimation.
We investigate the issue of bandwidth estimation in a nonparametric functional regression model with function-valued, continuous real-valued and discrete-valued regressors under the framework of unknown error density. Extending from the recent work of Shang (2013, Computational Statistics & Data Analysis), we approximate the unknown error density by a kernel density estimator of residuals, where the regression function is estimated by the functional Nadaraya-Watson estimator that admits mixed types of regressors. We derive a kernel likelihood and posterior density for the bandwidth parameters …
Multivariate nonnegative orthant data are real vectors bounded to the left by the null vector, and they can be continuous, discrete or mixed. We first review the recent relative variability indexes for multivariate nonnegative continuous and count distributions. As a prelude, the classification of two comparable distributions having the same mean vector is done through under-, equi- and over-variability with respect to the reference distribution. Multivariate associated kernel estimators are then reviewed with new proposals that can accommodate any nonnegative orthant dataset. We focus on band…
This study examines the optimal selections of bandwidth and semi-metric for a functional partial linear model. Our proposed method begins by estimating the unknown error density using a kernel density estimator of residuals, where the regression function, consisting of parametric and nonparametric components, can be estimated by functional principal component and functional Nadayara-Watson estimators. The estimation accuracy of the regression function and error density crucially depends on the optimal estimations of bandwidth and semi-metric. A Bayesian method is utilized to simultaneously est…
In the context of estimating local modes of a conditional density based on kernel density estimators, we show that existing bandwidth selection methods developed for kernel density estimation are unsuitable for mode estimation. We propose two methods to select bandwidths tailored for mode estimation in the regression setting. Numerical studies using synthetic data and a real-life data set are carried out to demonstrate the performance of the proposed methods in comparison with several well received bandwidth selection methods for density estimation.
Nonparametric regression models such as Bayesian Additive Regression Trees (BART) can be useful in fitting flexible functions of a set of covariates to a response, while accounting for nonlinearities and interactions. However, they are often cumbersome to interpret. Breaking down the function into additive components, if appropriate, could simplify the interpretation and improve the utility of the model. On the other hand, establishing nonadditivity can be useful in determining the need for individualized predictions and treatment selection. Testing additivity of single covariates in nonparame…
Bayesian models based on the Dirichlet process and other stick-breaking priors have been proposed as core ingredients for clustering, topic modeling, and other unsupervised learning tasks. Prior specification is, however, relatively difficult for such models, given that their flexibility implies that the consequences of prior choices are often relatively opaque. Moreover, these choices can have a substantial effect on posterior inferences. Thus, considerations of robustness need to go hand in hand with nonparametric modeling. In the current paper, we tackle this challenge by exploiting the fac…
Bayesian models based on the Dirichlet process and other stick-breaking priors have been proposed as core ingredients for clustering, topic modeling, and other unsupervised learning tasks. However, due to the flexibility of these models, the consequences of prior choices can be opaque. And so prior specification can be relatively difficult. At the same time, prior choice can have a substantial effect on posterior inferences. Thus, considerations of robustness need to go hand in hand with nonparametric modeling. In the current paper, we tackle this challenge by exploiting the fact that variatio…
We study location-scale mixture priors for nonparametric statistical problems, including multivariate regression, density estimation and classification. We show that a rate-adaptive procedure can be obtained if the prior is properly constructed. In particular, we show that adaptation is achieved if a kernel mixture prior on a regression function is constructed using a Gaussian kernel, an inverse gamma bandwidth, and Gaussian mixing weights.
We propose a new, two-step empirical Bayes-type of approach for neural networks. We show in context of the nonparametric regression model that the procedure (up to a logarithmic factor) provides optimal recovery of the underlying functional parameter of interest and provides Bayesian credible sets with frequentist coverage guarantees. The approach requires fitting the neural network only once, hence it is substantially faster than Bootstrapping type approaches. We demonstrate the applicability of our method over synthetic data, observing good estimation properties and reliable uncertainty quan…
Bayesian simultaneous estimation of nonparametric quantile curves is a challenging problem, requiring a flexible and robust data model whilst satisfying the monotonicity or noncrossing constraints on the quantiles. This paper presents the use of the pyramid quantile regression method in the spline regression setting. In high dimensional problems, the choice of the pyramid locations becomes crucial for a robust parameter estimation. In this work we derive the optimal {pyramid locations which then allows us to propose an efficient} adaptive block-update MCMC scheme for posterior computation. Sim…
We propose nonparametric estimation of divergence measures between continuous distributions. Our approach is based on a plug-in kernel- type estimators of density functions. We give the uniform in bandwidth consistency for the proposal estimators. As a consequence, their asymp- totic 100% confidence intervals are also provided.
Suppose that $Y = ψ(X_1, \ldots, X_p)$, where $(X_1,\ldots, X_p)^\top$ are random inputs, $Y$ is the output, and $ψ(\cdot)$ is an unknown link function. The Sobol indices gauge the sensitivity of each $X$ against $Y$ by estimating the regression curve's variability between them. In this paper, we estimate these curves with a kernel-based method. The method allows to estimate the first order indices when the link between the independent and dependent variables is unknown. The kernel-based methods need a bandwidth to average the observations. For finite samples, the cross-validation method is fa…
We propose a flexible nonparametric regression method for ultrahigh-dimensional data. As a first step, we propose a fast screening method based on the favored smoothing bandwidth of the marginal local constant regression. Then, an iterative procedure is developed to recover both the important covariates and the regression function. Theoretically, we prove that the favored smoothing bandwidth based screening possesses the model selection consistency property. Simulation studies as well as real data analysis show the competitive performance of the new procedure.
Over the last decade, big data have poured into econometrics, demanding new statistical methods for analysing high-dimensional data and complex non-linear relationships. A common approach for addressing dimensionality issues relies on the use of static graphical structures for extracting the most significant dependence interrelationships between the variables of interest. Recently, Bayesian nonparametric techniques have become popular for modelling complex phenomena in a flexible and efficient manner, but only few attempts have been made in econometrics. In this paper, we provide an innovative…
In nonparametric regression problems involving multiple predictors, there is typically interest in estimating an anisotropic multivariate regression surface in the important predictors while discarding the unimportant ones. Our focus is on defining a Bayesian procedure that leads to the minimax optimal rate of posterior contraction (up to a log factor) adapting to the unknown dimension and anisotropic smoothness of the true surface. We propose such an approach based on a Gaussian process prior with dimension-specific scalings, which are assigned carefully-chosen hyperpriors. We additionally sh…
We consider nonparametric Bayesian estimation inference using a rescaled smooth Gaussian field as a prior for a multidimensional function. The rescaling is achieved using a Gamma variable and the procedure can be viewed as choosing an inverse Gamma bandwidth. The procedure is studied from a frequentist perspective in three statistical settings involving replicated observations (density estimation, regression and classification). We prove that the resulting posterior distribution shrinks to the distribution that generates the data at a speed which is minimax-optimal up to a logarithmic factor, …
Consider binary observations whose response probability is an unknown smooth function of a set of covariates. Suppose that a prior on the response probability function is induced by a Gaussian process mapped to the unit interval through a link function. In this paper we study consistency of the resulting posterior distribution. If the covariance kernel has derivatives up to a desired order and the bandwidth parameter of the kernel is allowed to take arbitrarily small values, we show that the posterior distribution is consistent in the $L_1$-distance. As an auxiliary result to our proofs, we sh…
We propose completely nonparametric methodology to investigate location-scale modelling of two-component mixture cure models, where the responses of interest are only indirectly observable due to the presence of censoring and the presence of so-called long-term survivors that are always censored. We use covariate-localized nonparametric estimators, which depend on a bandwidth sequence, to propose an estimator of the error distribution function that has not been considered before in the literature. When this bandwidth belongs to a certain range of undersmoothing bandwidths, the asymptotic distr…
We consider the problem of bandwidth selection by cross-validation from a sequential point of view in a nonparametric regression model. Having in mind that in applications one often aims at estimation, prediction and change detection simultaneously, we investigate that approach for sequential kernel smoothers in order to base these tasks on a single statistic. We provide uniform weak laws of large numbers and weak consistency results for the cross-validated bandwidth. Extensions to weakly dependent error terms are discussed as well. The errors may be α-mixing or L2-near epoch dependent, which …
Cross-validation is a well-known and widely used bandwidth selection method in nonparametric regression estimation. However, this technique has two remarkable drawbacks: (i) the large variability of the selected bandwidths, and (ii) the inability to provide results in a reasonable time for very large sample sizes. To overcome these problems, bagging cross-validation bandwidths are analyzed in this paper. This approach consists in computing the cross-validation bandwidths for a finite number of subsamples and then rescaling the averaged smoothing parameters to the original sample size. Under a …
Reconstruction of sets from a random sample of points intimately related to them is the goal of set estimation theory. Within this context, a particular problem is the one related with the reconstruction of density level sets and specifically, those ones with a high probability content, namely highest density regions. We define highest density regions for directional data and provide a plug-in estimator, based on kernel smoothing. A suitable bootstrap bandwidth selector is provided for the practical implementation of the proposal. An extensive simulation study shows the performance of the plug…
Nonparametric kernel density and local polynomial regression estimators are very popular in Statistics, Economics, and many other disciplines. They are routinely employed in applied work, either as part of the main empirical analysis or as a preliminary ingredient entering some other estimation or inference procedure. This article describes the main methodological and numerical features of the software package nprobust, which offers an array of estimation and inference procedures for nonparametric kernel-based density and local polynomial regression methods, implemented in both the R and Stata…
Carnegie Mellon University（卡内基梅隆大学）；University of Rome（罗马大学）
We derive nonparametric confidence intervals for the eigenvalues of the Hessian at modes of a density estimate. This provides information about the strength and shape of modes and can also be used as a significance test. We use a data-splitting approach in which potential modes are identified using the first half of the data and inference is done with the second half of the data. To get valid confidence sets for the eigenvalues, we use a bootstrap based on an elementary-symmetric-polynomial (ESP) transformation. This leads to valid bootstrap confidence sets regardless of any multiplicities in …
The kedd package providing additional smoothing techniques to the R statistical system. Although various packages on the Comprehensive R Archive Network (CRAN) provide functions useful to nonparametric statistics, kedd aims to serve as a central location for more specifically of a nonparametric functions and data sets. The current feature set of the package can be split in four main categories: compute the convolutions and derivatives of a kernel function, compute the kernel estimators for a density of probability and its derivatives, computing the bandwidth selectors with different methods, d…
Carnegie Mellon University（卡内基梅隆大学）；University of Chicago（芝加哥大学）
We present algorithms for nonparametric regression in settings where the data are obtained sequentially. While traditional estimators select bandwidths that depend upon the sample size, for sequential data the effective sample size is dynamically changing. We propose a linear time algorithm that adjusts the bandwidth for each new data point, and show that the estimator achieves the optimal minimax rate of convergence. We also propose the use of online expert mixing algorithms to adapt to unknown smoothness of the regression function. We provide simulations that confirm the theoretical results,…
In this paper, we establish uniform-in-bandwidth limit laws of the logarithm for nonparametric Inverse Probability of Censoring Weighted (I.P.C.W.) estimators of the multivariate regression function under random censorship. A similar result is deduced for estimators of the conditional distribution function. The uniform-in-bandwidth consistency for estimators of the conditional density and the conditional hazard rate functions are also derived from our main result. Moreover, the logarithm laws we establish are shown to yield almost sure simultaneous asymptotic confidence bands for the functions…
This paper introduces a data-adaptive non-parametric approach for the estimation of time-varying spectral densities from nonstationary time series. Time-varying spectral densities are commonly estimated by local kernel smoothing. The performance of these nonparametric estimators, however, depends crucially on the smoothing bandwidths that need to be specified in both time and frequency direction. As an alternative and extension to traditional bandwidth selection methods, we propose an iterative algorithm for constructing localized smoothing kernels data-adaptively. The main idea, inspired by t…
Nonparametric estimators of a regression function with circular response and Rd-valued predictor are considered in this work. Local polynomial type estimators are proposed and studied. Expressions for their asymptotic biases and variances are derived, and some guidelines to select asymptotically local optimal bandwidth matrices are also given. The finite sample behavior of the proposed estimators is assessed through simulations and their performance is also illustrated with a real data set.
Density-based clustering relies on the idea of linking groups to some specific features of the probability distribution underlying the data. The reference to a true, yet unknown, population structure allows to frame the clustering problem in a standard inferential setting, where the concept of ideal population clustering is defined as the partition induced by the true density function. The nonparametric formulation of this approach, known as modal clustering, draws a correspondence between the groups and the domains of attraction of the density modes. Operationally, a nonparametric density est…
In a wide range of modern applications, we observe a large number of time series rather than only a single one. It is often natural to suppose that there is some group structure in the observed time series. When each time series is modelled by a nonparametric regression equation, one may in particular assume that the observed time series can be partitioned into a small number of groups whose members share the same nonparametric regression function. We develop a bandwidth-free clustering method to estimate the unknown group structure from the data. More precisely speaking, we construct multisca…
Conditional density estimation generalizes regression by modeling a full density f(yjx) rather than only the expected value E(yjx). This is important for many tasks, including handling multi-modality and generating prediction intervals. Though fundamental and widely applicable, nonparametric conditional density estimators have received relatively little attention from statisticians and little or none from the machine learning community. None of that work has been applied to greater than bivariate data, presumably due to the computational difficulty of data-driven bandwidth selection. We descri…
This study examines a nonparametric inference on a stationary Lévy-driven Ornstein-Uhlenbeck (OU) process $X = (X_{t})_{t \geq 0}$ with a compound Poisson subordinator. We propose a new spectral estimator for the Lévy measure of the Lévy-driven OU process $X$ under macroscopic observations. We also derive, for the estimator, multivariate central limit theorems over a finite number of design points, and high-dimensional central limit theorems in the case wherein the number of design points increases with an increase in the sample size. Built on these asymptotic results, we develop methods to co…
Continuous treatments (e.g., doses) arise often in practice, but many available causal effect estimators are limited by either requiring parametric models for the effect curve, or by not allowing doubly robust covariate adjustment. We develop a novel kernel smoothing approach that requires only mild smoothness assumptions on the effect curve, and still allows for misspecification of either the treatment density or outcome regression. We derive asymptotic properties and give a procedure for data-driven bandwidth selection. The methods are illustrated via simulation and in a study of the effect …
There is a long-standing debate in the statistical, epidemiological and econometric fields as to whether nonparametric estimation that uses data-adaptive methods, like machine learning algorithms in model fitting, confer any meaningful advantage over simpler, parametric approaches in real-world, finite sample estimation of causal effects. We address the question: when trying to estimate the effect of a treatment on an outcome, across a universe of reasonable data distributions, how much does the choice of nonparametric vs.~parametric estimation matter? Instead of answering this question with s…
In the context of regressing a response $Y$ on a predictor $X$, we consider estimating the local modes of the distribution of $Y$ given $X=x$ when $X$ is prone to measurement error. We propose two nonparametric estimation methods, with one based on estimating the joint density of $(X, Y)$ in the presence of measurement error, and the other built upon estimating the conditional density of $Y$ given $X=x$ using error-prone data. We study the asymptotic properties of each proposed mode estimator, and provide implementation details including the mean-shift algorithm for mode seeking and bandwidth …
Density level sets are mainly estimated using one of three methodologies: plug-in, excess mass, or a hybrid approach. The plug-in methods are based on replacing the unknown density by some nonparametric estimator, usually the kernel. Thus, the bandwidth selection is a fundamental problem from a practical point of view. Recently, specific selectors for level sets have been proposed. However, if some a priori information about the geometry of the level set is available, then excess mass algorithms can be useful. In this case, a density estimator is not necessary, and the problem of bandwidth sel…
In this paper, we consider the problem of estimating a conditional density in moderately large dimensions. Much more informative than regression functions, conditional densities are of main interest in recent methods, particularly in the Bayesian framework (studying the posterior distribution, finding its modes...). Considering a recently studied family of kernel estimators, we select a pointwise multivariate bandwidth by revisiting the greedy algorithm Rodeo (Regularisation Of Derivative Expectation Operator). The method addresses several issues: being greedy and computationally efficient by …
Nonparametric maximum likelihood estimation is intended to infer the unknown density distribution while making as few assumptions as possible. To alleviate the over parameterization in nonparametric data fitting, smoothing assumptions are usually merged into the estimation. In this paper a novel boosting-based method is introduced to the nonparametric estimation in univariate cases. We deduce the boosting algorithm by the second-order approximation of nonparametric log-likelihood. Gaussian kernel and smooth spline are chosen as weak learners in boosting to satisfy the smoothing assumptions. Si…
For the nonparametric regression models with covariates contaminated with normal measurement errors, this paper proposes an extrapolation algorithm to estimate the nonparametric regression functions. By applying the conditional expectation directly to the kernel-weighted least squares of the deviations between the local linear approximation and the observed responses, the proposed algorithm successfully bypasses the simulation step needed in the classical simulation extrapolation method, thus significantly reducing the computational time. It is noted that the proposed method also provides an e…
