We consider estimation of a step function $f$ from noisy observations of a deconvolution $ϕ*f$, where $ϕ$ is some bounded $L_1$-function. We use a penalized least squares estimator to reconstruct the signal $f$ from the observations, with penalty equal to the number of jumps of the reconstruction. Asymptotically, it is possible to correctly estimate the number of jumps with probability one. Given that the number of jumps is correctly estimated, we show that the corresponding parameter estimates of the jump locations and jump heights are $n^{-1/2}$ consistent and converge to a joint normal distribution with covariance structure depending on $ϕ$, and that this rate is minimax for bounded continuous kernels $ϕ$. As special case we obtain the asymptotic distribution of the least squares estimator in multiphase regression and generalisations thereof. In contrast to the results obtained for bounded $ϕ$, we show that for kernels with a singularity of order $O(| x|^{-α}),1/2<α<1$, a jump location can be estimated at a rate of $n^{-1/(3-2α)}$, which is again the minimax rate. We find that these rate do not depend on the spectral information of the operator rather on its localization properties in the time domain. Finally, it turns out that adaptive sampling does not improve the rate of convergence, in strict contrast to the case of direct regression.
We construct uniform and point-wise asymptotic confidence sets for the single edge in an otherwise smooth image function which are based on rotated differences of two one-sided kernel estimators. Using methods from M-estimation, we show consistency of the estimators of location, slope and height of the edge function and develop a uniform linearization of the contrast process. The uniform confidence bands then rely on a Gaussian approximation of the score process together with anti-concentration results for suprema of Gaussian processes, while point-wise bands are based on asymptotic normality.…
Online jump and kink detection in segmented linear regression: Statistical optimality meets computational efficiency
Institute for Mathematical Stochastics, University of Göttingen（哥廷根大学数学随机学研究所）；Cluster of Excellence “Multiscale Bioimaging: from Molecular Machines to Networks of Excitable Cells” (MBExC), University of Göttingen（哥廷根大学“多尺度生物成像：从分子机器到兴奋细胞网络”卓越集群）
In this paper we investigate an indirect regression model characterized by the Radon transformation. This model is useful for recovery of medical images obtained by computed tomography scans. The indirect regression function is estimated using a series estimator motivated by a spectral cut-off technique. Further, we investigate the empirical process of residuals from this regression, and show that it satsifies a functional central limit theorem.
In a recent paper Birke and Bissantz (2008) considered the problem of nonparametric estimation in inverse regression models with convolution-type operators. For multivariate predictors nonparametric methods suffer from the curse of dimensionality and we consider inverse regression models with the additional qualitative assumption of additivity. In these models several additive estimators are studied. In particular, we investigate estimators under the random design assumption which are applicable when observations are not available on a grid. Finally, we compare this estimator with the marginal…
We are presenting a method of linear regression based on Gram-Schmidt orthogonal projection that does not compute a pseudo-inverse matrix. This is useful when we want to make several regressions with random data vectors for simulation purposes.
Sliced inverse regression (Duan and Li [Ann. Statist. 19 (1991) 505-530], Li [J. Amer. Statist. Assoc. 86 (1991) 316-342]) is an appealing dimension reduction method for regression models with multivariate covariates. It has been extended by Ferré and Yao [Statistics 37 (2003) 475-488, Statist. Sinica 15 (2005) 665-683] and Hsing and Ren [Ann. Statist. 37 (2009) 726-755] to functional covariates where the whole trajectories of random functional covariates are completely observed. The focus of this paper is to develop sliced inverse regression for intermittently and sparsely measured longitudin…
We consider the statistical experiment of functional linear regression (FLR). Furthermore, we introduce a white noise model where one observes an Ito process, which contains the covariance operator of the corresponding FLR model in its construction. We prove asymptotic equivalence of FLR and this white noise model in LeCam's sense under known design distribution. Moreover, we show equivalence of FLR and an empirical version of the white noise model for finite sample sizes. As an application, we derive sharp minimax constants in the FLR model which are still valid in the case of unknown design …
This work deals with the ill-posed inverse problem of reconstructing a function $f$ given implicitly as the solution of $g = Af$, where $A$ is a compact linear operator with unknown singular values and known eigenfunctions. We observe the function $g$ and the singular values of the operator subject to Gaussian white noise with respective noise levels $\varepsilon$ and $σ$. We develop a minimax theory in terms of both noise levels and propose an orthogonal series estimator attaining the minimax rates. This estimator requires the optimal choice of a dimension parameter depending on certain chara…
The paper considers the problem of robust estimating a periodic function in a continuous time regression model with dependent disturbances given by a general square integrable semimartingale with unknown distribution. An example of such a noise is non-gaussian Ornstein-Uhlenbeck process with the Lévy process subordinator, which is used to model the financial Black-Scholes type markets with jumps. An adaptive model selection procedure, based on the weighted least square estimates, is proposed. Under general moment conditions on the noise distribution, sharp non-asymptotic oracle inequalities fo…
In this article we study the estimation of the location of jump points in the first derivative (referred to as kinks) of a regression function μin two random design models with different long-range dependent (LRD) structures. The method is based on the zero-crossing technique and makes use of high-order kernels. The rate of convergence of the estimator is contingent on the level of dependence and the smoothness of the regression function μ. In one of the models, the convergence rate is the same as the minimax rate for kink estimation in the fixed design scenario with i.i.d. errors which sugges…
In this paper we study the consistency of different bootstrap procedures for constructing confidence intervals (CIs) for the unique jump discontinuity (change-point) in an otherwise smooth regression function in a stochastic design setting. This problem exhibits nonstandard asymptotics and we argue that the standard bootstrap procedures in regression fail to provide valid confidence intervals for the change-point. We propose a version of smoothed bootstrap, illustrate its remarkable finite sample performance in our simulation study, and prove the consistency of the procedure. The $m$ out of $n…
This paper presents the nonparametric inference for nonlinear volatility functionals of general multivariate Itô semimartingales, in high-frequency and noisy setting. Pre-averaging and truncation enable simultaneous handling of noise and jumps. Second-order expansion reveals explicit biases and a pathway to bias correction. Estimators based on this framework achieve the optimal convergence rate. A class of stable central limit theorems are attained with estimable asymptotic covariance matrices. This paper form a basis for infill asymptotic results of, for example, the realized Laplace transfor…
We develop and justify methodology to consistently test for long-horizon return predictability based on realized variance. To accomplish this, we propose a parametric transaction-level model for the continuous-time log price process based on a pure jump point process. The model determines the returns and realized variance at any level of aggregation with properties shown to be consistent with the stylized facts in the empirical finance literature. Under our model, the long-memory parameter propagates unchanged from the transaction-level drift to the calendar-time returns and the realized varia…
We consider the problem of estimating an additive regression function in an inverse regres- sion model with a convolution type operator. A smooth backfitting procedure is developed and asymptotic normality of the resulting estimator is established. Compared to other meth- ods for the estimation in additive models the new approach neither requires observations on a regular grid nor the estimation of the joint density of the predictor. It is also demonstrated by means of a simulation study that the backfitting estimator outperforms the marginal in- tegration method at least by a factor two with …
When linear regression generates a relationship between a (dependent) scalar response and one or multiple independent variables, various datasets providing distinct graphical trends can develop resembling relationships based on the same statistical properties. Advanced statistical approaches, such as neural networks and machine learning methods, are of great necessity to process, characterize, and analyze these degenerate datasets. On the other hand, the accurate creation of purposedly degenerate datasets is essential to test new models in the research and education of applied statistics. In t…
We propose a new class of estimators of the multivariate response linear regression coefficient matrix that exploits the assumption that the response and predictors have a joint multivariate Normal distribution. This allows us to indirectly estimate the regression coefficient matrix through shrinkage estimation of the parameters of the inverse regression, or the conditional distribution of the predictors given the responses. We establish a convergence rate bound for estimators in our class and we study two examples. The first example estimator exploits an assumption that the inverse regression…
We study logistic regression with total variation penalty on the canonical parameter and show that the resulting estimator satisfies a sharp oracle inequality: the excess risk of the estimator is adaptive to the number of jumps of the underlying signal or an approximation thereof. In particular when there are finitely many jumps, and jumps up are sufficiently separated from jumps down, then the estimator converges with a parametric rate up to a logarithmic term $\log n / n$, provided the tuning parameter is chosen appropriately of order $1/ \sqrt n$. Our results extend earlier results for quad…
We consider Bayesian inference in inverse regression problems where the objective is to infer about unobserved covariates from observed responses and covariates. We establish posterior consistency of such unobserved covariates in Bayesian inverse regression problemsunder appropriate priors in a leave-one-out cross-validation setup. We relate this to posterior consistency of inverse reference distributions (Bhattacharya (2013)) for assessing model adequacy. We illustrate our theory and methods with various examples of Bayesian inverse regression, along with adequate simulation experiments.
The problem of estimating the baseline signal from multisample noisy curves is investigated. We consider the functional mixed effects model, and we suppose that the functional fixed effect belongs to the Besov class. This framework allows us to model curves that can exhibit strong irregularities, such as peaks or jumps for instance. The lower bound for the $L_2$ minimax risk is provided, as well as the upper bound of the minimax rate, that is derived by constructing a wavelet estimator for the functional fixed effect. Our work constitutes the first theoretical functional results in multisample…
A simple construction of adaptive confidence sets is proposed in isotonic, convex and unimodal regression. In univariate isotonic regression, the proposed confidence set enjoys uniform coverage over all non-decreasing regression functions. Furthermore, the diameter of the proposed confidence set automatically adapts to the unknown number of pieces of the true parameter, in the sense that the diameter is bounded from above by the minimax risk over the class of $k$-piecewise constant functions. The diameter of the confidence set is a simple increasing function of the number of jumps of the isoto…
Asymptotic equivalence in Le Cam's sense for nonparametric regression experiments is extended to the case of non-regular error densities, which have jump discontinuities at their endpoints. We prove asymptotic equivalence of such regression models and the observation of two independent Poisson point processes which contain the target curve as the support boundary of its intensity function. The intensity of the point processes is of order of the sample size $n$ and involves the jump sizes as well as the design density. The statistical model significantly differs from regression problems with Ga…
In this paper, we propose a dimension reduction model for spatially dependent variables. Namely, we investigate an extension of the \emph{inverse regression} method under strong mixing condition. This method is based on estimation of the matrix of covariance of the expectation of the explanatory given the dependent variable, called the \emph{inverse regression}. Then, we study, under strong mixing condition, the weak and strong consistency of this estimate, using a kernel estimate of the \emph{inverse regression}. We provide the asymptotic behaviour of this estimate. A spatial predictor based …
Sliced Inverse Regression (SIR) is an effective method for dimension reduction in high-dimensional regression problems. The original method, however, requires the inversion of the predictors covariance matrix. In case of collinearity between these predictors or small sample sizes compared to the dimension, the inversion is not possible and a regularization technique has to be used. Our approach is based on a Fisher Lecture given by R.D. Cook where it is shown that SIR axes can be interpreted as solutions of an inverse regression problem. In this paper, a Gaussian prior distribution is introduc…
Functional data analysis is a growing research field as more and more practical applications involve functional data. In this paper, we focus on the problem of regression and classification with functional predictors: the model suggested combines an efficient dimension reduction procedure [functional sliced inverse regression, first introduced by Ferré & Yao (Statistics, 37, 2003, 475)], for which we give a regularized version, with the accuracy of a neural network. Some consistency results are given and the method is successfully confronted to real-life data.
In this paper we consider a semiparametric regression model involving a $d$-dimensional quantitative explanatory variable $X$ and including a dimension reduction of $X$ via an index $β'X$. In this model, the main goal is to estimate the euclidean parameter $β$ and to predict the real response variable $Y$ conditionally to $X$. Our approach is based on sliced inverse regression (SIR) method and optimal quantization in $\mathbf{L}^p$-norm. We obtain the convergence of the proposed estimators of $β$ and of the conditional distribution. Simulation studies show the good numerical behavior of the pr…
We study the asymptotics for jump-penalized least squares regression aiming at approximating a regression function by piecewise constant functions. Besides conventional consistency and convergence rates of the estimates in $L^2([0,1))$ our results cover other metrics like Skorokhod metric on the space of càdlàg functions and uniform metrics on $C([0,1])$. We will show that these estimators are in an adaptive sense rate optimal over certain classes of "approximation spaces." Special cases are the class of functions of bounded variation (piecewise) Hölder continuous functions of order $0<α\le1$ …
In this paper we will discuss a procedure to improve the usual estimator of a linear functional of the unknown regression function in inverse nonparametric regression models. In Klaassen, Lee, and Ruymgaart (2001) it has been proved that this traditional estimator is not asymptotically efficient (in the sense of the Hájek - Le Cam convolution theorem) except, possibly, when the error distribution is normal. Since this estimator, however, is still root-n consistent a procedure in Bickel, Klaassen, Ritov, and Wellner (1993) applies to construct a modification which is asymptotically efficient. A…
In "Li, L. and Yin, X. (2008). Sliced Inverse Regression with Regularizations. Biometrics, 64(1):124--131" a ridge SIR estimator is introduced as the solution of a minimization problem and computed thanks to an alternating least-squares algorithm. This methodology reveals good performance in practice. In this note, we focus on the theoretical properties of the estimator. Is it shown that the minimization problem is degenerated in the sense that only two situations can occur: Either the ridge SIR estimator does not exist or it is zero.
We provide new theoretical results in the field of inverse regression methods for dimension reduction. Our approach is based on the study of some empirical processes that lie close to a certain dimension reduction subspace, called the central subspace. The study of these processes essentially includes weak convergence results and the consistency of some general bootstrap procedures. While such properties are used to obtain new results about sliced inverse regression, they mainly allow to define a natural family of methods for dimension reduction. First the estimation methods are shown to have …
In this article, we propose a general nonlinear sufficient dimension reduction (SDR) framework when both the predictor and response lie in some general metric spaces. We construct reproducing kernel Hilbert spaces whose kernels are fully determined by the distance functions of the metric spaces, then leverage the inherent structures of these spaces to define a nonlinear SDR framework. We adapt the classical sliced inverse regression of \citet{Li:1991} within this framework for the metric space data. We build the estimator based on the corresponding linear operators, and show it recovers the re…
Many machine learning tools for regression are based on recursive partitioning of the covariate space into smaller regions, where the regression function can be estimated locally. Among these, regression trees and their ensembles have demonstrated impressive empirical performance. In this work, we shed light on the machinery behind Bayesian variants of these methods. In particular, we study Bayesian regression histograms, such as Bayesian dyadic trees, in the simple regression case with just one predictor. We focus on the reconstruction of regression surfaces that are piecewise constant, where…
We consider a two-stage procedure (TSP) for estimating an inverse regression function at a given point, where isotonic regression is used at stage one to obtain an initial estimate and a local linear approximation in the vicinity of this estimate is used at stage two. We establish that the convergence rate of the second-stage estimate can attain the parametric $n^{1/2}$ rate. Furthermore, a bootstrapped variant of TSP (BTSP) is introduced and its consistency properties studied. This variant manages to overcome the slow speed of the convergence in distribution and the estimation of the derivati…
Suppose that $Y$ is a scalar and $X$ is a second-order stochastic process, where $Y$ and $X$ are conditionally independent given the random variables $ξ_1,...,ξ_p$ which belong to the closed span $L_X^2$ of $X$. This paper investigates a unified framework for the inverse regression dimension-reduction problem. It is found that the identification of $L_X^2$ with the reproducing kernel Hilbert space of $X$ provides a platform for a seamless extension from the finite- to infinite-dimensional settings. It also facilitates convenient computational algorithms that can be applied to a variety of mode…
Inverse problems, where in broad sense the task is to learn from the noisy response about some unknown function, usually represented as the argument of some known functional form, has received wide attention in the general scientific disciplines. How- ever, in mainstream statistics such inverse problem paradigm does not seem to be as popular. In this article we provide a brief overview of such problems from a statistical, particularly Bayesian, perspective. We also compare and contrast the above class of problems with the perhaps more statistically familiar inverse regression problems, arguing…
Predict a new response from a covariate is a challenging task in regression, which raises new question since the era of high-dimensional data. In this paper, we are interested in the inverse regression method from a theoretical viewpoint. Theoretical results have already been derived for the well-known linear model, but recently, the curse of dimensionality has increased the interest of practitioners and theoreticians into generalization of those results for various estimators, calibrated for the high-dimension context. To deal with high-dimensional data, inverse regression is used in this pap…
We consider an enlarged dimension reduction space in functional inverse regression. Our operator and functional analysis based approach facilitates a compact and rigorous formulation of the functional inverse regression problem. It also enables us to expand the possible space where the dimension reduction functions belong. Our formulation provides a unified framework so that the classical notions, such as covariance standardization, Mahalanobis distance, SIR and linear discriminant analysis, can be naturally and smoothly carried out in our enlarged space. This enlarged dimension reduction spac…
University of California Irvine（加州大学欧文分校）；London School of Economics and Political Science（伦敦政治经济学院）；North Carolina State University（北卡罗来纳州立大学）
An individualized decision rule (IDR) is a decision function that assigns each individual a given treatment based on his/her observed characteristics. Most of the existing works in the literature consider settings with binary or finitely many treatment options. In this paper, we focus on the continuous treatment setting and propose a jump interval-learning to develop an individualized interval-valued decision rule (I2DR) that maximizes the expected outcome. Unlike IDRs that recommend a single treatment, the proposed I2DR yields an interval of treatment options for each individual, making it mo…
We propose a new method for dimension reduction in regression using the first two inverse moments. We develop corresponding weighted chi-squared tests for the dimension of the regression. The proposed method considers linear combinations of Sliced Inverse Regression (SIR) and the method using a new candidate matrix which is designed to recover the entire inverse second moment subspace. The optimal combination may be selected based on the p-values derived from the dimension tests. Theoretically, the proposed method, as well as Sliced Average Variance Estimate (SAVE), are more capable of recover…
We provide here a framework to analyze the phase transition phenomenon of slice inverse regression (SIR), a supervised dimension reduction technique introduced by \cite{Li:1991}. Under mild conditions, the asymptotic ratio $ρ= \lim p/n$ is the phase transition parameter and the SIR estimator is consistent if and only if $ρ= 0$. When dimension $p$ is greater than $n$, we propose a diagonal thresholding screening SIR (DT-SIR) algorithm. This method provides us with an estimate of the eigen-space of the covariance matrix of the conditional expectation $var(\mathbf{E}[\boldsymbol{x}|y])$. The desi…
In this paper optimal experimental designs for inverse quadratic regression models are determined. We consider two different parameterizations of the model and investigate local optimal designs with respect to the $c$-, $D$- and $E$-criteria, which reflect various aspects of the precision of the maximum likelihood estimator for the parameters in inverse quadratic regression models. In particular it is demonstrated that for a sufficiently large design space geometric allocation rules are optimal with respect to many optimality criteria. Moreover, in numerous cases the designs with respect to th…
Uniform asymptotic confidence bands for a multivariate regression function in an inverse regression model with a convolution-type operator are constructed. The results are derived using strong approximation methods and a limit theorem for the supremum of a stationary Gaussian field over an increasing system of sets. As a particular application, asymptotic confidence bands for a time dependent regression function $f_t(x)$ ($x\in \mathbb {R}^d,t\in \mathbb {R}$) in a convolution-type inverse regression model are obtained. Finally, we demonstrate the practical feasibility of our proposed methods …
Online dimension reduction is a common method for high-dimensional streaming data processing. Online principal component analysis, online sliced inverse regression, online kernel principal component analysis and other methods have been studied in depth, but as far as we know, online supervised nonlinear dimension reduction methods have not been fully studied. In this article, an online kernel sliced inverse regression method is proposed. By introducing the approximate linear dependence condition and dictionary variable sets, we address the problem of increasing variable dimensions with the sam…
In the Bayesian approach, the a priori knowledge about the input of a mathematical model is described via a probability measure. The joint distribution of the unknown input and the data is then conditioned, using Bayes' formula, giving rise to the posterior distribution on the unknown input. In this setting we prove posterior consistency for nonlinear inverse problems: a sequence of data is considered, with diminishing fluctuations around a single truth and it is then of interest to show that the resulting sequence of posterior measures arising from this sequence of data concentrates around th…
When do nonparametric Bayesian procedures ``overfit''? To shed light on this question, we consider a binary regression problem in detail and establish frequentist consistency for a certain class of Bayes procedures based on hierarchical priors, called uniform mixture priors. These are defined as follows: let $ν$ be any probability distribution on the nonnegative integers. To sample a function $f$ from the prior $π^ν$, first sample $m$ from $ν$ and then sample $f$ uniformly from the set of step functions from $[0,1]$ into $[0,1]$ that have exactly $m$ jumps (i.e., sample all $m$ jump locations …
In this article, we propose a novel Bayesian multiple testing formulation for model and variable selection in inverse setups, judiciously embedding the idea of inverse reference distributions proposed by Bhattacharya (2013) in a mixture framework consisting of the competing models. We develop the theory and methods in the general context encompassing parametric and nonparametric competing models, dependent data, as well as misspecifications. Our investigation shows that asymptotically the multiple testing procedure almost surely selects the best possible inverse model that minimizes the minimu…
We consider the problem of supervised dimension reduction with a particular focus on extreme values of the target $Y\in\mathbb{R}$ to be explained by a covariate vector $X \in \mathbb{R}^p$. The general purpose is to define and estimate a projection on a lower dimensional subspace of the covariate space which is sufficient for predicting exceedances of the target above high thresholds. We propose an original definition of Tail Conditional Independence which matches this purpose. Inspired by Sliced Inverse Regression (SIR) methods, we develop a novel framework (TIREX, Tail Inverse Regression fo…
Consider a regression model with infinitely many parameters and time series errors. We are interested in choosing weights for averaging across generalized least squares (GLS) estimators obtained from a set of approximating models. However, GLS estimators, depending on the unknown inverse covariance matrix of the errors, are usually infeasible. We therefore construct feasible generalized least squares (FGLS) estimators using a consistent estimator of the unknown inverse matrix. Based on this inverse covariance matrix estimator and FGLS estimators, we develop a feasible autocovariance-corrected …
This paper investigates the large sample properties of the variance, weights, and risk of high-dimensional portfolios where the inverse of the covariance matrix of excess asset returns is estimated using a technique called nodewise regression. Nodewise regression provides a direct estimator for the inverse covariance matrix using the Least Absolute Shrinkage and Selection Operator (Lasso) of Tibshirani (1994) to estimate the entries of a sparse precision matrix. We show that the variance, weights, and risk of the global minimum variance portfolios and the Markowitz mean-variance portfolios are…
In the Bayesian literature on model comparison, Bayes factors play the leading role. In the classical statistical literature, model selection criteria are often devised used cross-validation ideas. Amalgamating the ideas of Bayes factor and cross-validation Geisser and Eddy (1979) created the pseudo-Bayes factor. The usage of cross-validation inculcates several theoretical advantages, computational simplicity and numerical stability in Bayes factors as the marginal density of the entire dataset is replaced with products of cross-validation densities of individual data points. However, the popu…
For multiple index models, it has recently been shown that the sliced inverse regression (SIR) is consistent for estimating the sufficient dimension reduction (SDR) space if and only if $ρ=\lim\frac{p}{n}=0$, where $p$ is the dimension and $n$ is the sample size. Thus, when $p$ is of the same or a higher order of $n$, additional assumptions such as sparsity must be imposed in order to ensure consistency for SIR. By constructing artificial response variables made up from top eigenvectors of the estimated conditional covariance matrix, we introduce a simple Lasso regression method to obtain an e…
