In this paper we investigate an indirect regression model characterized by the Radon transformation. This model is useful for recovery of medical images obtained by computed tomography scans. The indirect regression function is estimated using a series estimator motivated by a spectral cut-off technique. Further, we investigate the empirical process of residuals from this regression, and show that it satsifies a functional central limit theorem.
This manuscript is a supplemental document providing the omitted material for our paper entitled "Nonparametric kernel estimation of the probability density function of regression errors using estimated residuals" [arXiv:1010.0439]. The paper is submitted to Journal of Nonparametric Statistics.
This paper deals with the nonparametric density estimation of the regression error term assuming its independence with the covariate. The difference between the feasible estimator which uses the estimated residuals and the unfeasible one using the true residuals is studied. An optimal choice of the bandwidth used to estimate the residuals is given. We also study the asymptotic normality of the feasible kernel estimator and its rate-optimality.
We consider the residual empirical process in random design regression with long memory errors. We establish its limiting behaviour, showing that its rates of convergence are different from the rates of convergence for to the empirical process based on (unobserved) errors. Also, we study a residual empirical process with estimated parameters. Its asymptotic distribution can be used to construct Kolmogorov-Smirnov, Cramér-Smirnov-von Mises, or other goodness-of-fit tests. Theoretical results are justified by simulation studies.
We prove theorems about the Gaussian asymptotics of an empirical bridge built from linear model regressors with multiple regressor ordering. We study the testing of the hypothesis of a linear model for the components of a random vector: one of the components is a linear combination of the others up to an error that does not depend on the other components of the random vector. The results of observations of independent copies of a random vector are sequentially ordered in ascending order of several of its components. The result is a sequence of vectors of higher dimension, consisting of induced…
In a recent paper Birke and Bissantz (2008) considered the problem of nonparametric estimation in inverse regression models with convolution-type operators. For multivariate predictors nonparametric methods suffer from the curse of dimensionality and we consider inverse regression models with the additional qualitative assumption of additivity. In these models several additive estimators are studied. In particular, we investigate estimators under the random design assumption which are applicable when observations are not available on a grid. Finally, we compare this estimator with the marginal…
We are presenting a method of linear regression based on Gram-Schmidt orthogonal projection that does not compute a pseudo-inverse matrix. This is useful when we want to make several regressions with random data vectors for simulation purposes.
We propose a new class of semiparametric regression models of mean residual life for censored outcome data. The models, which enable us to estimate the expected remaining survival time and generalize commonly used mean residual life models, also conduct covariate dimension reduction. Using the geometric approaches in semiparametrics literature and the martingale properties with survival data, we propose a flexible inference procedure that relaxes the parametric assumptions on the dependence of mean residual life on covariates and how long a patient has lived. We show that the estimators for th…
Sliced inverse regression (Duan and Li [Ann. Statist. 19 (1991) 505-530], Li [J. Amer. Statist. Assoc. 86 (1991) 316-342]) is an appealing dimension reduction method for regression models with multivariate covariates. It has been extended by Ferré and Yao [Statistics 37 (2003) 475-488, Statist. Sinica 15 (2005) 665-683] and Hsing and Ren [Ann. Statist. 37 (2009) 726-755] to functional covariates where the whole trajectories of random functional covariates are completely observed. The focus of this paper is to develop sliced inverse regression for intermittently and sparsely measured longitudin…
The question whether a time series behaves as a random walk or as a station- ary process is an important and delicate problem, particularly arising in financial statistics, econometrics, and engineering. This paper studies the problem to detect sequentially that the error terms in a polynomial regression model no longer behave as a random walk but as a stationary process. We provide the asymptotic distribution theory for a monitoring procedure given by a control chart, i.e., a stopping time, which is related to a well known unit root test statistic calculated from sequentially updated residual…
We consider the statistical experiment of functional linear regression (FLR). Furthermore, we introduce a white noise model where one observes an Ito process, which contains the covariance operator of the corresponding FLR model in its construction. We prove asymptotic equivalence of FLR and this white noise model in LeCam's sense under known design distribution. Moreover, we show equivalence of FLR and an empirical version of the white noise model for finite sample sizes. As an application, we derive sharp minimax constants in the FLR model which are still valid in the case of unknown design …
We study the residual bootstrap (RB) method in the context of high-dimensional linear regression. Specifically, we analyze the distributional approximation of linear contrasts $c^{\top} (\hatβ_ρ-β)$, where $\hatβ_ρ$ is a ridge-regression estimator. When regression coefficients are estimated via least squares, classical results show that RB consistently approximates the laws of contrasts, provided that $p\ll n$, where the design matrix is of size $n\times p$. Up to now, relatively little work has considered how additional structure in the linear model may extend the validity of RB to the settin…
Quantile regression permits describing how quantiles of a scalar response variable depend on a set of predictors. Because a unique definition of multivariate quantiles is lacking, extending quantile regression to multivariate responses is somewhat complicated. In this paper, we describe a simple approach based on a two-step procedure: in the first step, quantile regression is applied to each response separately; in the second step, the joint distribution of the signs of the residuals is modeled through multinomial regression. The described approach does not require a multidimensional definitio…
We consider a heteroscedastic regression model in which some of the regression coefficients are zero but it is not known which ones. Penalized quantile regression is a useful approach for analyzing such data. By allowing different covariates to be relevant for modeling conditional quantile functions at different quantile levels, it provides a more complete picture of the conditional distribution of a response variable than mean regression. Existing work on penalized quantile regression has been mostly focused on point estimation. Although bootstrap procedures have recently been shown to be eff…
We consider a regression modeling of the quantiles of residual life, remaining lifetime at a specific time. We propose a smoothed induced version of the existing non-smooth estimating equations approaches for estimating regression parameters. The proposed estimating equations are smooth in regression parameters, so solutions can be readily obtained via standard numerical algorithms. Moreover, the smoothness in the proposed estimating equations enables one to obtain a robust sandwich-type covariance estimator of regression estimators aided by an efficient resampling method. To handle data subje…
Model diagnostics is an indispensable component of regression analysis, yet it is not well addressed in standard textbooks on generalized linear models. The lack of exposition is attributed to the fact that when outcome data are discrete, classical methods (e.g., Pearson/deviance residual analysis and goodness-of-fit tests) have limited utility in model diagnostics and treatment. This paper establishes a novel framework for model diagnostics of discrete data regression. Unlike the literature defining a single-valued quantity as the residual, we propose to use a function as a vehicle to retain …
This article considers nonparametric regression models with multivariate covariates and with responses missing at random. We estimate the regression function with a local polynomial smoother. The residual-based empirical distribution function that only uses complete cases, i.e. residuals that can actually be constructed from the data, is shown to be efficient in the sense of Hájek and Le Cam. In the proofs we derive, more generally, the efficient influence function for estimating an arbitrary linear functional of the error distribution; this covers the distribution function as a special case. …
We consider the problem of estimating an additive regression function in an inverse regres- sion model with a convolution type operator. A smooth backfitting procedure is developed and asymptotic normality of the resulting estimator is established. Compared to other meth- ods for the estimation in additive models the new approach neither requires observations on a regular grid nor the estimation of the joint density of the predictor. It is also demonstrated by means of a simulation study that the backfitting estimator outperforms the marginal in- tegration method at least by a factor two with …
When linear regression generates a relationship between a (dependent) scalar response and one or multiple independent variables, various datasets providing distinct graphical trends can develop resembling relationships based on the same statistical properties. Advanced statistical approaches, such as neural networks and machine learning methods, are of great necessity to process, characterize, and analyze these degenerate datasets. On the other hand, the accurate creation of purposedly degenerate datasets is essential to test new models in the research and education of applied statistics. In t…
We propose a new class of estimators of the multivariate response linear regression coefficient matrix that exploits the assumption that the response and predictors have a joint multivariate Normal distribution. This allows us to indirectly estimate the regression coefficient matrix through shrinkage estimation of the parameters of the inverse regression, or the conditional distribution of the predictors given the responses. We establish a convergence rate bound for estimators in our class and we study two examples. The first example estimator exploits an assumption that the inverse regression…
The length-biased Birnbaum-Saunders distribution is both useful and practical for environmental sciences. In this paper, we initially derive some new properties for the length-biased Birnbaum-Saunders distribution, showing that one of its parameters is the mode and that it is bimodal. We then introduce a new regression model based on this distribution. We implement use the maximum likelihood method for parameter estimation, approach interval estimation and consider three types of residuals. An elaborate Monte Carlo study is carried out for evaluating the performance of the likelihood-based est…
We consider Bayesian inference in inverse regression problems where the objective is to infer about unobserved covariates from observed responses and covariates. We establish posterior consistency of such unobserved covariates in Bayesian inverse regression problemsunder appropriate priors in a leave-one-out cross-validation setup. We relate this to posterior consistency of inverse reference distributions (Bhattacharya (2013)) for assessing model adequacy. We illustrate our theory and methods with various examples of Bayesian inverse regression, along with adequate simulation experiments.
Residual-based analysis is generally considered a cornerstone of statistical methodology. For a special case of indirect regression, we investigate the residual-based empirical distribution function and provide a uniform expansion of this estimator, which is also shown to be asymptotically most precise. This investigation naturally leads to a completely data-driven technique for selecting a regularization parameter used in our indirect regression function estimator. The resulting methodology is based on a smooth bootstrap of the model residuals. A simulation study demonstrates the effectivenes…
This work examines under what circumstances adaptivity for truncated SVD estimation can be achieved by an early stopping rule based on the smoothed residuals $ \| ( A A^{\top} )^{α/ 2} ( Y - A \hatμ^{( m )}) \|^{2} $. Lower and upper bounds for the risk are derived, which show that moderate smoothing of the residuals can be used to adapt over classes of signals with varying smoothness, while oversmoothing yields suboptimal convergence rates. The theoretical results are illustrated by Monte-Carlo simulations.
In this paper, we propose a dimension reduction model for spatially dependent variables. Namely, we investigate an extension of the \emph{inverse regression} method under strong mixing condition. This method is based on estimation of the matrix of covariance of the expectation of the explanatory given the dependent variable, called the \emph{inverse regression}. Then, we study, under strong mixing condition, the weak and strong consistency of this estimate, using a kernel estimate of the \emph{inverse regression}. We provide the asymptotic behaviour of this estimate. A spatial predictor based …
In several linear regression data sets, $Y (\in R)$ on ${\bf X} (\in R^p),$ visual comparisons of $L_1$ and $L_2$-residuals' plots indicate bad leverage cases. The phenomenon is confirmed theoretically by introducing Location Breakdown Point (LBP) of a functional $T$: any point where the derivative of $T$'s Influence Function either takes values at infinities or does not exist. Guidelines for the plots' visual comparisons as diagnostic are provided. The new tools used include E-matrix and suggest influence diagnostic RINFIN which measures the distance in the derivatives of $L_2$-residuals} at …
Sliced Inverse Regression (SIR) is an effective method for dimension reduction in high-dimensional regression problems. The original method, however, requires the inversion of the predictors covariance matrix. In case of collinearity between these predictors or small sample sizes compared to the dimension, the inversion is not possible and a regularization technique has to be used. Our approach is based on a Fisher Lecture given by R.D. Cook where it is shown that SIR axes can be interpreted as solutions of an inverse regression problem. In this paper, a Gaussian prior distribution is introduc…
Functional data analysis is a growing research field as more and more practical applications involve functional data. In this paper, we focus on the problem of regression and classification with functional predictors: the model suggested combines an efficient dimension reduction procedure [functional sliced inverse regression, first introduced by Ferré & Yao (Statistics, 37, 2003, 475)], for which we give a regularized version, with the accuracy of a neural network. Some consistency results are given and the method is successfully confronted to real-life data.
In this paper we consider a semiparametric regression model involving a $d$-dimensional quantitative explanatory variable $X$ and including a dimension reduction of $X$ via an index $β'X$. In this model, the main goal is to estimate the euclidean parameter $β$ and to predict the real response variable $Y$ conditionally to $X$. Our approach is based on sliced inverse regression (SIR) method and optimal quantization in $\mathbf{L}^p$-norm. We obtain the convergence of the proposed estimators of $β$ and of the conditional distribution. Simulation studies show the good numerical behavior of the pr…
In this paper we will discuss a procedure to improve the usual estimator of a linear functional of the unknown regression function in inverse nonparametric regression models. In Klaassen, Lee, and Ruymgaart (2001) it has been proved that this traditional estimator is not asymptotically efficient (in the sense of the Hájek - Le Cam convolution theorem) except, possibly, when the error distribution is normal. Since this estimator, however, is still root-n consistent a procedure in Bickel, Klaassen, Ritov, and Wellner (1993) applies to construct a modification which is asymptotically efficient. A…
In "Li, L. and Yin, X. (2008). Sliced Inverse Regression with Regularizations. Biometrics, 64(1):124--131" a ridge SIR estimator is introduced as the solution of a minimization problem and computed thanks to an alternating least-squares algorithm. This methodology reveals good performance in practice. In this note, we focus on the theoretical properties of the estimator. Is it shown that the minimization problem is degenerated in the sense that only two situations can occur: Either the ridge SIR estimator does not exist or it is zero.
We provide new theoretical results in the field of inverse regression methods for dimension reduction. Our approach is based on the study of some empirical processes that lie close to a certain dimension reduction subspace, called the central subspace. The study of these processes essentially includes weak convergence results and the consistency of some general bootstrap procedures. While such properties are used to obtain new results about sliced inverse regression, they mainly allow to define a natural family of methods for dimension reduction. First the estimation methods are shown to have …
This paper studies the residual empirical process of long- and short-memory time series regression models and establishes its uniform expansion under a general framework. The results are applied to the stochastic regression models and unstable autoregressive models. For the long-memory noise, it is shown that the limit distribution of the Kolmogorov-Smirnov test statistic studied in Ho and Hsing [Ann. Statist. 24 (1996) 992-1024] does not hold when the stochastic regression model includes an unknown intercept or when the characteristic polynomial of the unstable autoregressive model has a unit…
In this article, we propose a general nonlinear sufficient dimension reduction (SDR) framework when both the predictor and response lie in some general metric spaces. We construct reproducing kernel Hilbert spaces whose kernels are fully determined by the distance functions of the metric spaces, then leverage the inherent structures of these spaces to define a nonlinear SDR framework. We adapt the classical sliced inverse regression of \citet{Li:1991} within this framework for the metric space data. We build the estimator based on the corresponding linear operators, and show it recovers the re…
In this paper we consider a location model of the form $Y = m(X) + \varepsilon$, where $m(\cdot)$ is the unknown regression function, the error $\varepsilon$ is independent of the $p$-dimensional covariate $X$ and $E(\varepsilon)=0$. Given i.i.d. data $(X_1,Y_1),\ldots,(X_n,Y_n)$ and given an estimator $\hat m(\cdot)$ of the function $m(\cdot)$ (which can be parametric or nonparametric of nature), we estimate the distribution of the error term $\varepsilon$ by the empirical distribution of the residuals $Y_i-\hat m(X_i)$, $i=1,\ldots,n$. To approximate the distribution of this estimator, Koul …
We consider a two-stage procedure (TSP) for estimating an inverse regression function at a given point, where isotonic regression is used at stage one to obtain an initial estimate and a local linear approximation in the vicinity of this estimate is used at stage two. We establish that the convergence rate of the second-stage estimate can attain the parametric $n^{1/2}$ rate. Furthermore, a bootstrapped variant of TSP (BTSP) is introduced and its consistency properties studied. This variant manages to overcome the slow speed of the convergence in distribution and the estimation of the derivati…
Linear regression models are checked by a lack-of-fit (LOF) test to be sure that the model is at least approximatively true. In many practical cases data are sampled sequentially. Such a situation appears in industrial production when goods are produced one after the other. So it is of some interest to check the regression model sequentially. This can be done by recursive least squares residuals. A sequential LOF test can be based on the recursive residual partial sum process. In this paper we state the limit of the partial sum process of a triangular array of recursive residuals given a const…
Suppose that $Y$ is a scalar and $X$ is a second-order stochastic process, where $Y$ and $X$ are conditionally independent given the random variables $ξ_1,...,ξ_p$ which belong to the closed span $L_X^2$ of $X$. This paper investigates a unified framework for the inverse regression dimension-reduction problem. It is found that the identification of $L_X^2$ with the reproducing kernel Hilbert space of $X$ provides a platform for a seamless extension from the finite- to infinite-dimensional settings. It also facilitates convenient computational algorithms that can be applied to a variety of mode…
Inverse problems, where in broad sense the task is to learn from the noisy response about some unknown function, usually represented as the argument of some known functional form, has received wide attention in the general scientific disciplines. How- ever, in mainstream statistics such inverse problem paradigm does not seem to be as popular. In this article we provide a brief overview of such problems from a statistical, particularly Bayesian, perspective. We also compare and contrast the above class of problems with the perhaps more statistically familiar inverse regression problems, arguing…
Predict a new response from a covariate is a challenging task in regression, which raises new question since the era of high-dimensional data. In this paper, we are interested in the inverse regression method from a theoretical viewpoint. Theoretical results have already been derived for the well-known linear model, but recently, the curse of dimensionality has increased the interest of practitioners and theoreticians into generalization of those results for various estimators, calibrated for the high-dimension context. To deal with high-dimensional data, inverse regression is used in this pap…
We consider an enlarged dimension reduction space in functional inverse regression. Our operator and functional analysis based approach facilitates a compact and rigorous formulation of the functional inverse regression problem. It also enables us to expand the possible space where the dimension reduction functions belong. Our formulation provides a unified framework so that the classical notions, such as covariance standardization, Mahalanobis distance, SIR and linear discriminant analysis, can be naturally and smoothly carried out in our enlarged space. This enlarged dimension reduction spac…
We propose a new method for dimension reduction in regression using the first two inverse moments. We develop corresponding weighted chi-squared tests for the dimension of the regression. The proposed method considers linear combinations of Sliced Inverse Regression (SIR) and the method using a new candidate matrix which is designed to recover the entire inverse second moment subspace. The optimal combination may be selected based on the p-values derived from the dimension tests. Theoretically, the proposed method, as well as Sliced Average Variance Estimate (SAVE), are more capable of recover…
We provide here a framework to analyze the phase transition phenomenon of slice inverse regression (SIR), a supervised dimension reduction technique introduced by \cite{Li:1991}. Under mild conditions, the asymptotic ratio $ρ= \lim p/n$ is the phase transition parameter and the SIR estimator is consistent if and only if $ρ= 0$. When dimension $p$ is greater than $n$, we propose a diagonal thresholding screening SIR (DT-SIR) algorithm. This method provides us with an estimate of the eigen-space of the covariance matrix of the conditional expectation $var(\mathbf{E}[\boldsymbol{x}|y])$. The desi…
In this paper optimal experimental designs for inverse quadratic regression models are determined. We consider two different parameterizations of the model and investigate local optimal designs with respect to the $c$-, $D$- and $E$-criteria, which reflect various aspects of the precision of the maximum likelihood estimator for the parameters in inverse quadratic regression models. In particular it is demonstrated that for a sufficiently large design space geometric allocation rules are optimal with respect to many optimality criteria. Moreover, in numerous cases the designs with respect to th…
Uniform asymptotic confidence bands for a multivariate regression function in an inverse regression model with a convolution-type operator are constructed. The results are derived using strong approximation methods and a limit theorem for the supremum of a stationary Gaussian field over an increasing system of sets. As a particular application, asymptotic confidence bands for a time dependent regression function $f_t(x)$ ($x\in \mathbb {R}^d,t\in \mathbb {R}$) in a convolution-type inverse regression model are obtained. Finally, we demonstrate the practical feasibility of our proposed methods …
Online dimension reduction is a common method for high-dimensional streaming data processing. Online principal component analysis, online sliced inverse regression, online kernel principal component analysis and other methods have been studied in depth, but as far as we know, online supervised nonlinear dimension reduction methods have not been fully studied. In this article, an online kernel sliced inverse regression method is proposed. By introducing the approximate linear dependence condition and dictionary variable sets, we address the problem of increasing variable dimensions with the sam…
In this article, we propose a novel Bayesian multiple testing formulation for model and variable selection in inverse setups, judiciously embedding the idea of inverse reference distributions proposed by Bhattacharya (2013) in a mixture framework consisting of the competing models. We develop the theory and methods in the general context encompassing parametric and nonparametric competing models, dependent data, as well as misspecifications. Our investigation shows that asymptotically the multiple testing procedure almost surely selects the best possible inverse model that minimizes the minimu…
Proportional mean residual life model is studied for analysing survival data from the case-cohort design. To simultaneously estimate the regression parameters and the baseline mean residual life function, weighted estimating equations based on an inverse selection probability are proposed. The resulting regression coefficients estimates are shown to be consistent and asymptotic normal with easily estimated variance-covariance. Simulation studies show that the proposed estimators perform very well. An application to a real dataset from the South Welsh nickel refiners study is also given to illu…
