Shape constrained regression analysis has applications in dose-response modeling, environmental risk assessment, disease screening and many other areas. Incorporating the shape constraints can improve estimation efficiency and avoid implausible results. We propose two novel methods focusing on Bayesian monotone curve and surface estimation using Gaussian process projections. The first projects samples from an unconstrained prior, while the second projects samples from the Gaussian process posterior. Theory is developed on continuity of the projection, posterior consistency and rates of contraction. The second approach is shown to have an empirical Bayes justification and to lead to simple computation with good performance in finite samples. Our projection approach can be applied in other constrained function estimation problems including in multivariate settings.
We develop an adaptive monotone shrinkage estimator for regression models with the following characteristics: i) dense coefficients with small but important effects; ii) a priori ordering that indicates the probable predictive importance of the features. We capture both properties with an empirical Bayes estimator that shrinks coefficients monotonically with respect to their anticipated importance. This estimator can be rapidly computed using a version of Pool-Adjacent-Violators algorithm. We show that the proposed monotone shrinkage approach is competitive with the class of all Bayesian estim…
In many applications there is interest in estimating the relation between a predictor and an outcome when the relation is known to be monotone or otherwise constrained due to the physical processes involved. We consider one such application--inferring time-resolved aerosol concentration from a low-cost differential pressure sensor. The objective is to estimate a monotone function and make inference on the scaled first derivative of the function. We proposed Bayesian nonparametric monotone regression which uses a Bernstein polynomial basis to construct the regression function and puts a Dirichl…
In projection pursuit regression (PPR), an unknown response function is approximated by the sum of M "ridge functions," which are flexible functions of one-dimensional projections of a multivariate input space. Traditionally, optimization routines are used to estimate the projection directions and ridge functions via a sequential algorithm, and M is typically chosen via cross-validation. We introduce the first Bayesian version of PPR, which has the benefit of accurate uncertainty quantification. To learn the projection directions and ridge functions, we apply novel adaptations of methods used …
A local projection model is defined by a set of linear regressions that account for the associations between exogenous variables and an endogenous variable observed at different time points. While it is standard practice to separately estimate individual regressions using the ordinary least squares estimator, some recent studies treat a local projection model as a multivariate regression with correlated errors, i.e., seemingly unrelated regressions, and propose Bayesian and non-Bayesian methods to improve the estimation accuracy. However, it is not clear how and when this way of treatment of l…
Examples with bound information on the regression function and density abound in many real applications. We propose a novel approach for estimating such functions by incorporating the prior knowledge on the bounds. Specially, a Gaussian process is first imposed on the regression function whose posterior distribution is then projected onto the bounded space. The resulting projected measure is then used for inference. The projected sample path has closed form which facilitates efficient computations. In particular, our projection approach maintains a comparable computational efficiency with that…
Shape restrictions such as monotonicity on functions often arise naturally in statistical modeling. We consider a Bayesian approach to the problem of estimation of a monotone regression function and testing for monotonicity. We construct a prior distribution using piecewise constant functions. For estimation, a prior imposing monotonicity of the heights of these steps is sensible, but the resulting posterior is harder to analyze theoretically. We consider a ``projection-posterior'' approach, where a conjugate normal prior is used, but the monotonicity constraint is imposed on posterior samples…
We consider a Bayesian method for simultaneous quantile regression on a real variable. By monotone transformation, we can make both the response variable and the predictor variable take values in the unit interval. A representation of quantile function is given by a convex combination of two monotone increasing functions $ξ_1$ and $ξ_2$ not depending on the prediction variables. In a Bayesian approach, a prior is put on quantile functions by putting prior distributions on $ξ_1$ and $ξ_2$. The monotonicity constraint on the curves $ξ_1$ and $ξ_2$ are obtained through a spline basis expansion wi…
This paper studies the problem of testing whether a function is monotone from a nonparametric Bayesian perspective. Two new families of tests are constructed. The first uses constrained smoothing splines, together with a hierarchical stochastic-process prior that explicitly controls the prior probability of monotonicity. The second uses regression splines, together with two proposals for the prior over the regression coefficients. The finite-sample performance of the tests is shown via simulation to improve upon existing frequentist and Bayesian methods. The asymptotic properties of the Bayes …
We consider the nonparametric regression problem with multiple predictors and an additive error, where the regression function is assumed to be coordinatewise nondecreasing. We propose a Bayesian approach to make an inference on the multivariate monotone regression function, obtain the posterior contraction rate, and construct a universally consistent Bayesian testing procedure for multivariate monotonicity. To facilitate posterior analysis, we set aside the shape restrictions temporarily, and endow a prior on blockwise constant regression functions with heights independently normally distribu…
Gaussian mixtures are commonly used for modeling heavy-tailed error distributions in robust linear regression. Combining the likelihood of a multivariate robust linear regression model with a standard improper prior distribution yields an analytically intractable posterior distribution that can be sampled using a data augmentation algorithm. When the response matrix has missing entries, there are unique challenges to the application and analysis of the convergence properties of the algorithm. Conditions for geometric ergodicity are provided when the incomplete data have a "monotone" structure.…
This article deals with random projections applied as a data reduction technique for Bayesian regression analysis. We show sufficient conditions under which the entire $d$-dimensional distribution is approximately preserved under random projections by reducing the number of data points from $n$ to $k\in O(\operatorname{poly}(d/\varepsilon))$ in the case $n\gg d$. Under mild assumptions, we prove that evaluating a Gaussian likelihood function based on the projected data instead of the original data yields a $(1+O(\varepsilon))$-approximation in terms of the $\ell_2$ Wasserstein distance. Our ma…
Shape-constrained inference has wide applicability in bioassay, medicine, economics, risk assessment, and many other fields. Although there has been a large amount of work on monotone-constrained univariate curve estimation, multivariate shape-constrained problems are much more challenging, and fewer advances have been made in this direction. With a focus on monotone regression with multiple predictors, this current work proposes a projection approach to estimate a multiple monotone regression function. An initial unconstrained estimator -- such as a local polynomial estimator or spline estima…
Most common parametric families of copulas are totally ordered, and in many cases they are also positively or negatively regression dependent and therefore they lead to monotone regression functions, which makes them not suitable for dependence relationships that imply or suggest a non-monotone regression function. A gluing copula approach is proposed to decompose the underlying copula into totally ordered copulas that combined may lead to a non-monotone regression function.
We derive minimax generalized Bayes estimators of regression coefficients in the general linear model with spherically symmetric errors under invariant quadratic loss for the case of unknown scale. The class of estimators generalizes the class considered in Maruyama and Strawderman (2005) to include non-monotone shrinkage functions.
A primary goal of computer experiments is to reconstruct the function given by the computer code via scattered evaluations. Traditional isotropic Gaussian process models suffer from the curse of dimensionality, when the input dimension is relatively high given limited data points. Gaussian process models with additive correlation functions are scalable to dimensionality, but they are more restrictive as they only work for additive functions. In this work, we consider a projection pursuit model, in which the nonparametric part is driven by an additive Gaussian process regression. We choose the …
Fudan University（复旦大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
Pareto Front (PF) modeling is essential in decision making problems across all domains such as economics, medicine or engineering. In Operation Research literature, this task has been addressed based on multi-objective optimization algorithms. However, without learning models for PF, these methods cannot examine whether a new provided point locates on PF or not. In this paper, we reconsider the task from Data Mining perspective. A novel projection based active Gaussian process regression (P- aGPR) method is proposed for efficient PF modeling. First, P- aGPR chooses a series of projection space…
This paper presents a new variable selection approach integrated with Gaussian process (GP) regression. We consider a sparse projection of input variables and a general stationary covariance model that depends on the Euclidean distance between the projected features. The sparse projection matrix is considered as an unknown parameter. We propose a forward stagewise approach with embedded gradient descent steps to co-optimize the parameter with other covariance parameters based on the maximization of a non-convex marginal likelihood function with a concave sparsity penalty, and some convergence …
We consider so-called univariate unlinked (sometimes ``decoupled,'' or ``shuffled'') regression when the unknown regression curve is monotone. In standard monotone regression, one observes a pair $(X,Y)$ where a response $Y$ is linked to a covariate $X$ through the model $Y= m_0(X) + ε$, with $m_0$ the (unknown) monotone regression function and $ε$ the unobserved error (assumed to be independent of $X$). In the unlinked regression setting one gets only to observe a vector of realizations from both the response $Y$ and from the covariate $X$ where now $Y \stackrel{d}{=} m_0(X) + ε$. There is no…
In observational studies, potential confounders may distort the causal relationship between an exposure and an outcome. However, under some conditions, a causal dose-response curve can be recovered using the G-computation formula. Most classical methods for estimating such curves when the exposure is continuous rely on restrictive parametric assumptions, which carry significant risk of model misspecification. Nonparametric estimation in this context is challenging because in a nonparametric model these curves cannot be estimated at regular rates. Many available nonparametric estimators are sen…
An efficient monotone data augmentation (MDA) algorithm is proposed for missing data imputation for incomplete multivariate nonnormal data that may contain variables of different types, and are modeled by a sequence of regression models including the linear, binary logistic, multinomial logistic, proportional odds, Poisson, negative binomial, skew-normal, skew-t regressions or a mixture of these models. The MDA algorithm is applied to the sensitivity analyses of longitudinal trials with nonignorable dropout using the controlled pattern imputations that assume the treatment effect reduces or di…
In many problems, a sensible estimator of a possibly multivariate monotone function may itself fail to be monotone. We study the correction of such an estimator obtained via projection onto the space of functions monotone over a finite grid in the domain. We demonstrate that this corrected estimator has no worse supremal estimation error than the initial estimator, and that analogously corrected confidence bands contain the true function whenever the initial bands do, at no loss to average or maximal band width. Additionally, we demonstrate that the corrected estimator is uniformly asymptotica…
In this paper we propose a new test of heteroscedasticity for parametric regression models and partial linear regression models in high dimensional settings. When the dimension of covariates is large, existing tests of heteroscedasticity perform badly due to the \curse of dimensionality". To attack this problem, we construct a test of heteroscedasticity by using a projection-based empirical process. We study the asymptotic properties of the test statistic under the null hypothesis and alternative hypotheses. It is shown that the test can detect local alternatives departure from the null hypoth…
The goal of nonparametric regression is to recover an underlying regression function from noisy observations, under the assumption that the regression function belongs to a pre-specified infinite dimensional function space. In the online setting, when the observations come in a stream, it is generally computationally infeasible to refit the whole model repeatedly. There are as of yet no methods that are both computationally efficient and statistically rate-optimal. In this paper, we propose an estimator for online nonparametric regression. Notably, our estimator is an empirical risk minimizer …
A longstanding problem of existing empirical process-based tests for regressions is that when the number of covariates is greater than one, they either have no tractable limiting null distributions or are not omnibus. To attack this problem, we in this paper propose a projection-based adaptive-to-model approach. When the hypothetical model is parametric single-index, the method can fully utilize the dimension reduction model structure under the null hypothesis as if the covariate were one-dimensional such that the martingale transformation-based test can be asymptotically distribution-free. Fu…
We consider the problem of constructing pointwise confidence intervals in the multiple isotonic regression model. Recently, [HZ19] obtained a pointwise limit distribution theory for the so-called block max-min and min-max estimators [FLN17] in this model, but inference remains a difficult problem due to the nuisance parameter in the limit distribution that involves multiple unknown partial derivatives of the true regression function. In this paper, we show that this difficult nuisance parameter can be effectively eliminated by taking advantage of information beyond point estimates in the block…
We propose a new modeling and estimation approach to select the optimal treatment regime from different options through constructing a robust estimating equation. The method is protected against misspecification of the propensity score model, the outcome regression model for the non-treated group, or the potential non-monotonic treatment difference model. Our method also allows residual errors to depend on covariates. A single index structure is incorporated to facilitate the nonparametric estimation of the treatment difference. We then identify the optimal treatment through maximizing the val…
Depth notions in regression have been systematically proposed and examined in Zuo (2018). One of the prominent advantages of notion of depth is that it can be directly utilized to introduce median-type deepest estimating functionals (or estimators in empirical distribution case) for location or regression parameters in a multi-dimensional setting. Regression depth shares the advantage. Depth induced deepest estimating functionals are expected to inherit desirable and inherent robustness properties ( e.g. bounded maximum bias and influence function and high breakdown point) as their univariate …
Zuo (2019) (Z19) addressed the computation of the projection regression depth (PRD) and its induced median (the maximum depth estimator). Z19 achieved the exact computation of PRD via a modified version of regular univariate sample median, which resulted in the loss of invariance of PRD and the equivariance of depth induced median. This article achieves the exact computation without scarifying the invariance of PRD and the equivariance of the regression median. Z19 also addressed the approximate computation of PRD induced median, the naive algorithm in Z19 is very slow. This article modifies t…
We consider the problem of estimating the regression function in functional linear regression models by proposing a new type of projection estimators which combine dimension reduction and thresholding. The introduction of a threshold rule allows to get consistency under broad assumptions as well as minimax rates of convergence under additional regularity hypotheses. We also consider the particular case of Sobolev spaces generated by the trigonometric basis which permits to get easily mean squared error of prediction as well as estimators of the derivatives of the regression function. We prove …
The issue of honesty in constructing confidence sets arises in nonparametric regression. While optimal rate in nonparametric estimation can be achieved and utilized to construct sharp confidence sets, severe degradation of confidence level often happens after estimating the degree of smoothness. Similarly, for high-dimensional regression, oracle inequalities for sparse estimators could be utilized to construct sharp confidence sets. Yet the degree of sparsity itself is unknown and needs to be estimated, causing the honesty problem. To resolve this issue, we develop a novel method to construct …
We consider a general monotone regression estimation where we allow for independent and dependent regressors. We propose a modification of the classical isotonic least squares estimator and establish its rate of convergence for the integrated $L_1$-loss function. The methodology captures the shape of the data without assuming additivity or a parametric form for the regression function. Furthermore, the degree of smoothing is chosen automatically and no auxiliary tuning is required for the theoretical analysis. Some simulations and two real data illustrations complement the study of the propose…
