In this paper we will discuss a procedure to improve the usual estimator of a linear functional of the unknown regression function in inverse nonparametric regression models. In Klaassen, Lee, and Ruymgaart (2001) it has been proved that this traditional estimator is not asymptotically efficient (in the sense of the Hájek - Le Cam convolution theorem) except, possibly, when the error distribution is normal. Since this estimator, however, is still root-n consistent a procedure in Bickel, Klaassen, Ritov, and Wellner (1993) applies to construct a modification which is asymptotically efficient. A self-contained proof of the asymptotic efficiency is included.
This paper concerns the estimation of the regression function at a given point in nonparametric heteroscedastic models with Gaussian noise or with noise having unknown distribution. In the two cases an asymptotically efficient kernel estimator is constructed for the minimax absolute error risk.
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper, 2007, for estimating a unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk, i.e. the asymptotic quadratic risk for this procedure coincides with the Pinsker constant which gives a sharp lower bound for the quadratic risk over all possible estimators.
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper, 2007, for estimating an unknown nonparametric regression. %\cite{GaPe1}. We prove that this procedure is asymptotically efficient for a quadratic risk, i.e. the asymptotic quadratic risk for this procedure coincides with the Pinsker constant which gives a sharp lower bound for the quadratic risk over all possible estimates
We consider Bayesian inference in inverse regression problems where the objective is to infer about unobserved covariates from observed responses and covariates. We establish posterior consistency of such unobserved covariates in Bayesian inverse regression problemsunder appropriate priors in a leave-one-out cross-validation setup. We relate this to posterior consistency of inverse reference distributions (Bhattacharya (2013)) for assessing model adequacy. We illustrate our theory and methods with various examples of Bayesian inverse regression, along with adequate simulation experiments.
The optimum quality that can be asymptotically achieved in the estimation of a probability p using inverse binomial sampling is addressed. A general definition of quality is used in terms of the risk associated with a loss function that satisfies certain assumptions. It is shown that the limit superior of the risk for p asymptotically small has a minimum over all (possibly randomized) estimators. This minimum is achieved by certain non-randomized estimators. The model includes commonly used quality criteria as particular cases. Applications to the non-asymptotic regime are discussed considerin…
In this paper, we propose a dimension reduction model for spatially dependent variables. Namely, we investigate an extension of the \emph{inverse regression} method under strong mixing condition. This method is based on estimation of the matrix of covariance of the expectation of the explanatory given the dependent variable, called the \emph{inverse regression}. Then, we study, under strong mixing condition, the weak and strong consistency of this estimate, using a kernel estimate of the \emph{inverse regression}. We provide the asymptotic behaviour of this estimate. A spatial predictor based …
In this paper we investigate an indirect regression model characterized by the Radon transformation. This model is useful for recovery of medical images obtained by computed tomography scans. The indirect regression function is estimated using a series estimator motivated by a spectral cut-off technique. Further, we investigate the empirical process of residuals from this regression, and show that it satsifies a functional central limit theorem.
Sliced Inverse Regression (SIR) is an effective method for dimension reduction in high-dimensional regression problems. The original method, however, requires the inversion of the predictors covariance matrix. In case of collinearity between these predictors or small sample sizes compared to the dimension, the inversion is not possible and a regularization technique has to be used. Our approach is based on a Fisher Lecture given by R.D. Cook where it is shown that SIR axes can be interpreted as solutions of an inverse regression problem. In this paper, a Gaussian prior distribution is introduc…
Functional data analysis is a growing research field as more and more practical applications involve functional data. In this paper, we focus on the problem of regression and classification with functional predictors: the model suggested combines an efficient dimension reduction procedure [functional sliced inverse regression, first introduced by Ferré & Yao (Statistics, 37, 2003, 475)], for which we give a regularized version, with the accuracy of a neural network. Some consistency results are given and the method is successfully confronted to real-life data.
In a recent paper Birke and Bissantz (2008) considered the problem of nonparametric estimation in inverse regression models with convolution-type operators. For multivariate predictors nonparametric methods suffer from the curse of dimensionality and we consider inverse regression models with the additional qualitative assumption of additivity. In these models several additive estimators are studied. In particular, we investigate estimators under the random design assumption which are applicable when observations are not available on a grid. Finally, we compare this estimator with the marginal…
We consider the sparse high-dimensional linear regression model $Y=Xb+ε$ where $b$ is a sparse vector. For the Bayesian approach to this problem, many authors have considered the behavior of the posterior distribution when, in truth, $Y=Xβ+ε$ for some given $β$. There have been numerous results about the rate at which the posterior distribution concentrates around $β$, but few results about the shape of that posterior distribution. We propose a prior distribution for $b$ such that the marginal posterior distribution of an individual coordinate $b_i$ is asymptotically normal centered around an …
In this paper we consider a semiparametric regression model involving a $d$-dimensional quantitative explanatory variable $X$ and including a dimension reduction of $X$ via an index $β'X$. In this model, the main goal is to estimate the euclidean parameter $β$ and to predict the real response variable $Y$ conditionally to $X$. Our approach is based on sliced inverse regression (SIR) method and optimal quantization in $\mathbf{L}^p$-norm. We obtain the convergence of the proposed estimators of $β$ and of the conditional distribution. Simulation studies show the good numerical behavior of the pr…
We are presenting a method of linear regression based on Gram-Schmidt orthogonal projection that does not compute a pseudo-inverse matrix. This is useful when we want to make several regressions with random data vectors for simulation purposes.
In "Li, L. and Yin, X. (2008). Sliced Inverse Regression with Regularizations. Biometrics, 64(1):124--131" a ridge SIR estimator is introduced as the solution of a minimization problem and computed thanks to an alternating least-squares algorithm. This methodology reveals good performance in practice. In this note, we focus on the theoretical properties of the estimator. Is it shown that the minimization problem is degenerated in the sense that only two situations can occur: Either the ridge SIR estimator does not exist or it is zero.
We provide new theoretical results in the field of inverse regression methods for dimension reduction. Our approach is based on the study of some empirical processes that lie close to a certain dimension reduction subspace, called the central subspace. The study of these processes essentially includes weak convergence results and the consistency of some general bootstrap procedures. While such properties are used to obtain new results about sliced inverse regression, they mainly allow to define a natural family of methods for dimension reduction. First the estimation methods are shown to have …
In this paper, we address the challenging problem of selecting tuning parameters for high-dimensional sparse regression. We propose a simple and computationally efficient method, called path thresholding (PaTh), that transforms any tuning parameter-dependent sparse regression algorithm into an asymptotically tuning-free sparse regression algorithm. More specifically, we prove that, as the problem size becomes large (in the number of variables and in the number of observations), PaTh performs accurate sparse regression, under appropriate conditions, without specifying a tuning parameter. In fin…
In this article, we propose a general nonlinear sufficient dimension reduction (SDR) framework when both the predictor and response lie in some general metric spaces. We construct reproducing kernel Hilbert spaces whose kernels are fully determined by the distance functions of the metric spaces, then leverage the inherent structures of these spaces to define a nonlinear SDR framework. We adapt the classical sliced inverse regression of \citet{Li:1991} within this framework for the metric space data. We build the estimator based on the corresponding linear operators, and show it recovers the re…
Sliced inverse regression (Duan and Li [Ann. Statist. 19 (1991) 505-530], Li [J. Amer. Statist. Assoc. 86 (1991) 316-342]) is an appealing dimension reduction method for regression models with multivariate covariates. It has been extended by Ferré and Yao [Statistics 37 (2003) 475-488, Statist. Sinica 15 (2005) 665-683] and Hsing and Ren [Ann. Statist. 37 (2009) 726-755] to functional covariates where the whole trajectories of random functional covariates are completely observed. The focus of this paper is to develop sliced inverse regression for intermittently and sparsely measured longitudin…
We consider a two-stage procedure (TSP) for estimating an inverse regression function at a given point, where isotonic regression is used at stage one to obtain an initial estimate and a local linear approximation in the vicinity of this estimate is used at stage two. We establish that the convergence rate of the second-stage estimate can attain the parametric $n^{1/2}$ rate. Furthermore, a bootstrapped variant of TSP (BTSP) is introduced and its consistency properties studied. This variant manages to overcome the slow speed of the convergence in distribution and the estimation of the derivati…
This paper concerns statistical inference for the components of a high-dimensional regression parameter despite possible endogeneity of each regressor. Given a first-stage linear model for the endogenous regressors and a second-stage linear model for the dependent variable, we develop a novel adaptation of the parametric one-step update to a generic second-stage estimator. We provide conditions under which the scaled update is asymptotically normal. We then introduce a two-stage Lasso procedure and show that the second-stage Lasso estimator satisfies the aforementioned conditions. Using these …
Suppose that $Y$ is a scalar and $X$ is a second-order stochastic process, where $Y$ and $X$ are conditionally independent given the random variables $ξ_1,...,ξ_p$ which belong to the closed span $L_X^2$ of $X$. This paper investigates a unified framework for the inverse regression dimension-reduction problem. It is found that the identification of $L_X^2$ with the reproducing kernel Hilbert space of $X$ provides a platform for a seamless extension from the finite- to infinite-dimensional settings. It also facilitates convenient computational algorithms that can be applied to a variety of mode…
Inverse problems, where in broad sense the task is to learn from the noisy response about some unknown function, usually represented as the argument of some known functional form, has received wide attention in the general scientific disciplines. How- ever, in mainstream statistics such inverse problem paradigm does not seem to be as popular. In this article we provide a brief overview of such problems from a statistical, particularly Bayesian, perspective. We also compare and contrast the above class of problems with the perhaps more statistically familiar inverse regression problems, arguing…
Predict a new response from a covariate is a challenging task in regression, which raises new question since the era of high-dimensional data. In this paper, we are interested in the inverse regression method from a theoretical viewpoint. Theoretical results have already been derived for the well-known linear model, but recently, the curse of dimensionality has increased the interest of practitioners and theoreticians into generalization of those results for various estimators, calibrated for the high-dimension context. To deal with high-dimensional data, inverse regression is used in this pap…
We consider an enlarged dimension reduction space in functional inverse regression. Our operator and functional analysis based approach facilitates a compact and rigorous formulation of the functional inverse regression problem. It also enables us to expand the possible space where the dimension reduction functions belong. Our formulation provides a unified framework so that the classical notions, such as covariance standardization, Mahalanobis distance, SIR and linear discriminant analysis, can be naturally and smoothly carried out in our enlarged space. This enlarged dimension reduction spac…
We provide here a framework to analyze the phase transition phenomenon of slice inverse regression (SIR), a supervised dimension reduction technique introduced by \cite{Li:1991}. Under mild conditions, the asymptotic ratio $ρ= \lim p/n$ is the phase transition parameter and the SIR estimator is consistent if and only if $ρ= 0$. When dimension $p$ is greater than $n$, we propose a diagonal thresholding screening SIR (DT-SIR) algorithm. This method provides us with an estimate of the eigen-space of the covariance matrix of the conditional expectation $var(\mathbf{E}[\boldsymbol{x}|y])$. The desi…
We propose a new method for dimension reduction in regression using the first two inverse moments. We develop corresponding weighted chi-squared tests for the dimension of the regression. The proposed method considers linear combinations of Sliced Inverse Regression (SIR) and the method using a new candidate matrix which is designed to recover the entire inverse second moment subspace. The optimal combination may be selected based on the p-values derived from the dimension tests. Theoretically, the proposed method, as well as Sliced Average Variance Estimate (SAVE), are more capable of recover…
In this paper, we are concerned with nonparametric estimation of the multivariate regression function in the presence of right censored data. More precisely, we propose a statistic that is shown to be asymptotically normally distributed under the additive assumption, and that could be used to test for additivity in the censored regression setting.
In this paper optimal experimental designs for inverse quadratic regression models are determined. We consider two different parameterizations of the model and investigate local optimal designs with respect to the $c$-, $D$- and $E$-criteria, which reflect various aspects of the precision of the maximum likelihood estimator for the parameters in inverse quadratic regression models. In particular it is demonstrated that for a sufficiently large design space geometric allocation rules are optimal with respect to many optimality criteria. Moreover, in numerous cases the designs with respect to th…
We consider the statistical experiment of functional linear regression (FLR). Furthermore, we introduce a white noise model where one observes an Ito process, which contains the covariance operator of the corresponding FLR model in its construction. We prove asymptotic equivalence of FLR and this white noise model in LeCam's sense under known design distribution. Moreover, we show equivalence of FLR and an empirical version of the white noise model for finite sample sizes. As an application, we derive sharp minimax constants in the FLR model which are still valid in the case of unknown design …
We consider nonsynchronous sampling of parameterized stochastic regression models, which contain stochastic differential equations. Constructing a quasi-likelihood function, we prove that the quasi-maximum likelihood estimator and the Bayes type estimator are consistent and asymptotically mixed normal when the sampling frequency of the nonsynchronous data becomes large.
Uniform asymptotic confidence bands for a multivariate regression function in an inverse regression model with a convolution-type operator are constructed. The results are derived using strong approximation methods and a limit theorem for the supremum of a stationary Gaussian field over an increasing system of sets. As a particular application, asymptotic confidence bands for a time dependent regression function $f_t(x)$ ($x\in \mathbb {R}^d,t\in \mathbb {R}$) in a convolution-type inverse regression model are obtained. Finally, we demonstrate the practical feasibility of our proposed methods …
The paper deals with asymptotic properties of the adaptive procedure proposed in the author paper (2007) for estimation of unknown nonparametric regression. We prove that this procedure is asymptotically efficient for a quadratic risk. It means that the asymptotic quadratic risk for this procedure coincides with a sharp lower bound.
In the Bayesian approach, the a priori knowledge about the input of a mathematical model is described via a probability measure. The joint distribution of the unknown input and the data is then conditioned, using Bayes' formula, giving rise to the posterior distribution on the unknown input. In this setting we prove posterior consistency for nonlinear inverse problems: a sequence of data is considered, with diminishing fluctuations around a single truth and it is then of interest to show that the resulting sequence of posterior measures arising from this sequence of data concentrates around th…
Online dimension reduction is a common method for high-dimensional streaming data processing. Online principal component analysis, online sliced inverse regression, online kernel principal component analysis and other methods have been studied in depth, but as far as we know, online supervised nonlinear dimension reduction methods have not been fully studied. In this article, an online kernel sliced inverse regression method is proposed. By introducing the approximate linear dependence condition and dictionary variable sets, we address the problem of increasing variable dimensions with the sam…
In an instrumental variable model, the score statistic can be bounded for any alternative in parts of the parameter space. These regions involve a constraint on the first-stage regression coefficients and the reduced-form covariance matrix. Consequently, the Lagrange Multiplier test can have power close to size, despite being efficient under standard asymptotics. This information loss limits the power of conditional tests which use only the Anderson-Rubin and the score statistic. The conditional quasi-likelihood ratio test also suffers severe losses because it can be bounded for any alternativ…
This paper studies the identification and estimation of weighted average derivatives of conditional location functionals including conditional mean and conditional quantiles in settings where either the outcome variable or a regressor is interval-valued. Building on Manski and Tamer (2002) who study nonparametric bounds for mean regression with interval data, we characterize the identified set of weighted average derivatives of regression functions. Since the weighted average derivatives do not rely on parametric specifications for the regression functions, the identified set is well-defined w…
We show that nonparametric regression is asymptotically equivalent in Le Cam's sense with a sequence of Gaussian white noise experiments as the number of observations tends to infinity. We propose a general constructive framework based on approximation spaces, which permits to achieve asymptotic equivalence even in the cases of multivariate and random design.
This work deals with the ill-posed inverse problem of reconstructing a function $f$ given implicitly as the solution of $g = Af$, where $A$ is a compact linear operator with unknown singular values and known eigenfunctions. We observe the function $g$ and the singular values of the operator subject to Gaussian white noise with respective noise levels $\varepsilon$ and $σ$. We develop a minimax theory in terms of both noise levels and propose an orthogonal series estimator attaining the minimax rates. This estimator requires the optimal choice of a dimension parameter depending on certain chara…
In this article, we propose a novel Bayesian multiple testing formulation for model and variable selection in inverse setups, judiciously embedding the idea of inverse reference distributions proposed by Bhattacharya (2013) in a mixture framework consisting of the competing models. We develop the theory and methods in the general context encompassing parametric and nonparametric competing models, dependent data, as well as misspecifications. Our investigation shows that asymptotically the multiple testing procedure almost surely selects the best possible inverse model that minimizes the minimu…
This paper establishes asymptotic results for the maximum likelihood and restricted maximum likelihood (REML) estimators of the parameters in the nested error regression model for clustered data when both of the number of independent clusters and the cluster sizes (the number of observations in each cluster) go to infinity. Under very mild conditions, the estimators are shown to be asymptotically normal with an elegantly structured covariance matrix. There are no restrictions on the rate at which the cluster size tends to infinity but it turns out that we need to treat within cluster parameter…
A long-standing problem in the construction of asymptotically correct confidence bands for a regression function $m(x)=E[Y|X=x]$, where $Y$ is the response variable influenced by the covariate $X$, involves the situation where $Y$ values may be missing at random, and where the selection probability, the density function $f(x)$ of $X$, and the conditional variance of $Y$ given $X$ are all completely unknown. This can be particularly more complicated in nonparametric situations. In this paper we propose a new kernel-type regression estimator and study the limiting distribution of the properly no…
We consider the problem of supervised dimension reduction with a particular focus on extreme values of the target $Y\in\mathbb{R}$ to be explained by a covariate vector $X \in \mathbb{R}^p$. The general purpose is to define and estimate a projection on a lower dimensional subspace of the covariate space which is sufficient for predicting exceedances of the target above high thresholds. We propose an original definition of Tail Conditional Independence which matches this purpose. Inspired by Sliced Inverse Regression (SIR) methods, we develop a novel framework (TIREX, Tail Inverse Regression fo…
We prove some efficient inference results concerning estimation of a Ornstein-Uhlenbeck regression model, which is driven by a non-Gaussian stable Levy process and where the output process is observed at high-frequency over a fixed time period. Local asymptotics for the likelihood function is presented, followed by a way to construct an asymptotically efficient estimator through a suboptimal yet very simple preliminary estimator.
Consider a regression model with infinitely many parameters and time series errors. We are interested in choosing weights for averaging across generalized least squares (GLS) estimators obtained from a set of approximating models. However, GLS estimators, depending on the unknown inverse covariance matrix of the errors, are usually infeasible. We therefore construct feasible generalized least squares (FGLS) estimators using a consistent estimator of the unknown inverse matrix. Based on this inverse covariance matrix estimator and FGLS estimators, we develop a feasible autocovariance-corrected …
This paper investigates the large sample properties of the variance, weights, and risk of high-dimensional portfolios where the inverse of the covariance matrix of excess asset returns is estimated using a technique called nodewise regression. Nodewise regression provides a direct estimator for the inverse covariance matrix using the Least Absolute Shrinkage and Selection Operator (Lasso) of Tibshirani (1994) to estimate the entries of a sparse precision matrix. We show that the variance, weights, and risk of the global minimum variance portfolios and the Markowitz mean-variance portfolios are…
It is proved that nonparametric autoregression is asymptotically equivalent in the sense of Le Cam's deficiency distance to nonparametric regression with random design as well as with regular nonrandom design.
This paper provides ANOVA inference for nonparametric local polynomial regression (LPR) in analogy with ANOVA tools for the classical linear regression model. A surprisingly simple and exact local ANOVA decomposition is established, and a local R-squared quantity is defined to measure the proportion of local variation explained by fitting LPR. A global ANOVA decomposition is obtained by integrating local counterparts, and a global R-squared and a symmetric projection matrix are defined. We show that the proposed projection matrix is asymptotically idempotent and asymptotically orthogonal to it…
This paper considers linear panel data models where the dependence of the regressors and the unobservables is modelled through a factor structure. The asymptotic setting is such that the number of time periods and the sample size both go to infinity. Non-strong factors are allowed and the number of factors can grow to infinity with the sample size. We study a class of two-step estimators of the regression coefficients. In the first step, factors and factor loadings are estimated. Then, the second step corresponds to the panel regression of the outcome on the regressors and the estimates of the…
In the Bayesian literature on model comparison, Bayes factors play the leading role. In the classical statistical literature, model selection criteria are often devised used cross-validation ideas. Amalgamating the ideas of Bayes factor and cross-validation Geisser and Eddy (1979) created the pseudo-Bayes factor. The usage of cross-validation inculcates several theoretical advantages, computational simplicity and numerical stability in Bayes factors as the marginal density of the entire dataset is replaced with products of cross-validation densities of individual data points. However, the popu…
For multiple index models, it has recently been shown that the sliced inverse regression (SIR) is consistent for estimating the sufficient dimension reduction (SDR) space if and only if $ρ=\lim\frac{p}{n}=0$, where $p$ is the dimension and $n$ is the sample size. Thus, when $p$ is of the same or a higher order of $n$, additional assumptions such as sparsity must be imposed in order to ensure consistency for SIR. By constructing artificial response variables made up from top eigenvectors of the estimated conditional covariance matrix, we introduce a simple Lasso regression method to obtain an e…
