This paper studies nonparametric regression with long memory (LRD) errors and predictors. First, we formulate general conditions which guarantee the standard rate of convergence for a nonparametric kernel estimator. Second, we calculate the Mean Integrated Squared Error (MISE). In particular, we show that LRD of errors may influence MISE. On the other hand, an estimator for a shape function is typically not influenced by LRD in errors. Finally, we investigate properties of a data-driven bandwidth choice. We show that Averaged Squared Error (ASE) is a good approximation of MISE, however, this is not the case for a cross-validation criterion.
Most data sets comprise of measurements on continuous and categorical variables. In regression and classification Statistics literature, modeling high-dimensional mixed predictors has received limited attention. In this paper we study the general regression problem of inferring on a variable of interest based on high dimensional mixed continuous and binary predictors. The aim is to find a lower dimensional function of the mixed predictor vector that contains all the modeling information in the mixed predictors for the response, which can be either continuous or categorical. The approach we pro…
For analyzing unit-level multivariate data in small area estimation, we consider the multivariate nested error regression model (MNER) and provide the empirical best linear unbiased predictor (EBLUP) of a small area characteristic based on second-order unbiased and consistent estimators of the `within' and `between' multivariate components of variance. The second-order approximation of the mean squared error (MSE) matrix of the EBLUP and its unbiased estimator are derived in closed forms. The confidence interval with second-order accuracy is also provided analytically.
In this article we study the estimation of the location of jump points in the first derivative (referred to as kinks) of a regression function μin two random design models with different long-range dependent (LRD) structures. The method is based on the zero-crossing technique and makes use of high-order kernels. The rate of convergence of the estimator is contingent on the level of dependence and the smoothness of the regression function μ. In one of the models, the convergence rate is the same as the minimax rate for kink estimation in the fixed design scenario with i.i.d. errors which sugges…
We consider nonparametric estimation of a regression function for a situation where precisely measured predictors are used to estimate the regression curve for coarsened, that is, less precise or contaminated predictors. Specifically, while one has available a sample $(W_1,Y_1),...,(W_n,Y_n)$ of independent and identically distributed data, representing observations with precisely measured predictors, where $\mathrm{E}(Y_i|W_i)=g(W_i)$, instead of the smooth regression function $g$, the target of interest is another smooth regression function $m$ that pertains to predictors $X_i$ that are nois…
Regression problems are traditionally analyzed via univariate characteristics like the regression function, scale function and marginal density of regression errors. These characteristics are useful and informative whenever the association between the predictor and the response is relatively simple. More detailed information about the association can be provided by the conditional density of the response given the predictor. For the first time in the literature, this article develops the theory of minimax estimation of the conditional density for regression settings with fixed and random desig…
Minimax $L_2$ risks for high-dimensional nonparametric regression are derived under two sparsity assumptions: (1) the true regression surface is a sparse function that depends only on $d=O(\log n)$ important predictors among a list of $p$ predictors, with $\log p=o(n)$; (2) the true regression surface depends on $O(n)$ predictors but is an additive function where each additive component is sparse but may contain two or more interacting predictors and may have a smoothness level different from other components. For either modeling assumption, a practicable extension of the widely used Bayesian …
We consider the double functional nonparametric regression model $Y=r(X)+ε$, where the response variable $Y$ is Hilbert space-valued and the covariate $X$ takes values in a pseudometric space. The data satisfy an ergodicity criterion which dates back to Laib and Louani (2010) and are arranged in a triangular array. So our model also applies to samples obtained from spatial processes, e.g., stationary random fields indexed by the regular lattice $\mathbb{Z}^N$ for some $N\in\mathbb{N}_+$. We consider a kernel estimator of the Nadaraya--Watson type for the regression operator $r$ and study its l…
We consider covariate adjusted regression (CAR), a regression method for situations where predictors and response are observed after being distorted by a multiplicative factor. The distorting factors are unknown functions of an observable covariate, where one specific distorting function is associated with each predictor or response. The dependence of both response and predictors on the same confounding covariate may alter the underlying regression relation between undistorted but unobserved predictors and response. We consider a class of highly flexible adjustment methods for parameter estima…
We consider a multivariate linear response regression in which the number of responses and predictors is large and comparable with the number of observations, and the rank of the matrix of regression coefficients is assumed to be small. We study the distribution of singular values for the matrix of regression coefficients and for the matrix of predicted responses. For both matrices, it is found that the limit distribution of the largest singular value is a rescaling of the Tracy-Widom distribution. Based on this result, we suggest algorithms for the model rank selection and compare them with t…
For the class of stationary Gaussian long memory processes, we study some properties of the least-squares predictor of X_{n+1} based on (X_n, ..., X_1). The predictor is obtained by projecting X_{n+1} onto the finite past and the coefficients of the predictor are estimated on the same realisation. First we prove moment bounds for the inverse of the empirical covariance matrix. Then we deduce an asymptotic expression of the mean-squared error. In particular we give a relation between the number of terms used to estimate the coefficients and the number of past terms used for prediction, which en…
We study a functional linear regression model that deals with functional responses and allows for both functional covariates and high-dimensional vector covariates. The proposed model is flexible and nests several functional regression models in the literature as special cases. Based on the theory of reproducing kernel Hilbert spaces (RKHS), we propose a penalized least squares estimator that can accommodate functional variables observed on discrete sample points. Besides a conventional smoothness penalty, a group Lasso-type penalty is further imposed to induce sparsity in the high-dimensional…
This paper develops a method to construct uniform confidence bands for a nonparametric regression function where a predictor variable is subject to a measurement error. We allow for the distribution of the measurement error to be unknown, but assume the availability of validation data or repeated measurements on the latent predictor variable. The proposed confidence band builds on the deconvolution kernel estimation and a novel application of the multiplier bootstrap method. We establish asymptotic validity of the proposed confidence band. To our knowledge, this is the first paper to derive as…
In this manuscript, we study quantile regression in partial functional linear model where response is scalar and predictors include both scalars and multiple functions. Wavelet basis are adopted to better approximate functional slopes while effectively detect local features. The sparse group lasso penalty is imposed to select important functional predictors while capture shared information among them. The estimation problem can be reformulated into a standard second-order cone program and then solved by an interior point method. We also give a novel algorithm by using alternating direction met…
Groupwise Predictor Envelope Models for Multivariate Linear Regression
The Institute of Statistical Mathematics（统计数学研究所）；Graduate School of Mathematics, Kyushu University（九州大学大学院数理学府）；Faculty of Mathematics, Kyushu University（九州大学数学部）
In various applications of regression analysis, in addition to errors in the dependent observations also errors in the predictor variables play a substantial role and need to be incorporated in the statistical modeling process. In this paper we consider a nonparametric measurement error model of Berkson type with fixed design regressors and centered random errors, which is in contrast to much existing work in which the predictors are taken as random observations with random noise. Based on an estimator that takes the error in the predictor into account and on a suitable Gaussian approximation,…
In prediction problems with more predictors than observations, it can sometimes be helpful to use a joint probability model, $π(Y,X)$, rather than a purely conditional model, $π(Y \mid X)$, where $Y$ is a scalar response variable and $X$ is a vector of predictors. This approach is motivated by the fact that in many situations the marginal predictor distribution $π(X)$ can provide useful information about the parameter values governing the conditional regression. However, under very mild misspecification, this marginal distribution can also lead conditional inferences astray. Here, we explore t…
A regression model is proposed for the analysis of an ordinal response variable depending on a set of multiple covariates containing ordinal and potentially other variables. The proportional odds model (McCullagh (1980)) is used for the ordinal response, and constrained maximum likelihood estimation is used to account for the ordinality of covariates. Ordinal predictors are coded by dummy variables. The parameters associated to the categories of the ordinal predictor(s) are constrained, enforcing them to be monotonic (isotonic or antitonic). A decision rule is introduced for classifying the or…
We study tail risk dynamics in high-frequency financial markets and their connection with trading activity and market uncertainty. We introduce a dynamic extreme value regression model accommodating both stationary and local unit-root predictors to appropriately capture the time-varying behaviour of the distribution of high-frequency extreme losses. To characterize trading activity and market uncertainty, we consider several volatility and liquidity predictors, and propose a two-step adaptive $L_1$-regularized maximum likelihood estimator to select the most appropriate ones. We establish the o…
Compositional data are common in many fields, both as outcomes and predictor variables. The inventory of models for the case when both the outcome and predictor variables are compositional is limited and the existing models are difficult to interpret, due to their use of complex log-ratio transformations. We develop a transformation-free linear regression model where the expected value of the compositional outcome is expressed as a single Markov transition from the compositional predictor. Our approach is based on generalized method of moments thereby not requiring complete specification of da…
We propose a principal components regression method based on maximizing a joint pseudo-likelihood for responses and predictors. Our method uses both responses and predictors to select linear combinations of the predictors relevant for the regression, thereby addressing an oft-cited deficiency of conventional principal components regression. The proposed estimator is shown to be consistent in a wide range of settings, including ones with non-normal and dependent observations; conditions on the first and second moments suffice if the number of predictors ($p$) is fixed and the number of observat…
In regression analysis of multivariate data, it is tacitly assumed that response and predictor variables in each observed response-predictor pair correspond to the same entity or unit. In this paper, we consider the situation of "permuted data" in which this basic correspondence has been lost. Several recent papers have considered this situation without further assumptions on the underlying permutation. In applications, the latter is often to known to have additional structure that can be leveraged. Specifically, we herein consider the common scenario of "sparsely permuted data" in which only …
Soil Organic Carbon (SOC) estimation is crucial to manage both natural and anthropic ecosystems and has recently been put under the magnifying glass after the Paris agreement 2016 due to its relationship with greenhouse gas. Statistical applications have dominated the SOC stock mapping at regional scale so far. However, the community has hardly ever attempted to implement Quantile Regression (QR) to spatially predict the SOC distribution. In this contribution, we test QR to estimate SOC stock (0-30 $cm$ depth) in the agricultural areas of a highly variable semi-arid region (Sicily, Italy, arou…
Generalized linear and additive models are very efficient regression tools but the selection of relevant terms becomes difficult if higher order interactions are needed. In contrast, tree-based methods also known as recursive partitioning are explicitly designed to model a specific form of interaction but with their focus on interaction tend to neglect the main effects. The method proposed here focusses on the main effects of categorical predictors by using tree type methods to obtain clusters. In particular when the predictor has many categories one wants to know which of the categories have …
Functional linear regression is an important topic in functional data analysis. It is commonly assumed that samples of the functional predictor are independent realizations of an underlying stochastic process, and are observed over a grid of points contaminated by i.i.d. measurement errors. In practice, however, the dynamical dependence across different curves may exist and the parametric assumption on the error covariance structure could be unrealistic. In this paper, we consider functional linear regression with serially dependent observations of the functional predictor, when the contaminat…
The Australian National University（澳大利亚国立大学）；Nanjing University of Science and Technology（南京理工大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；The University of Sydney（悉尼大学）
We study online linear regression problems in a distributed setting, where the data is spread over a network. In each round, each network node proposes a linear predictor, with the objective of fitting the \emph{network-wide} data. It then updates its predictor for the next round according to the received local feedback and information received from neighboring nodes. The predictions made at a given node are assessed through the notion of regret, defined as the difference between their cumulative network-wide square errors and those of the best off-line network-wide linear predictor. Various s…
In the functional linear regression model, many methods have been proposed and studied to estimate the slope function while the functional predictor was observed in the entire domain. However, works on functional linear regression models with partially observed trajectories have received less attention. In this paper, to fill the literature gap we consider the scenario where individual functional predictor may be observed only on part of the domain. Depending on whether measurement error is presented in functional predictors, two methods are developed, one is based on linear functionals of the…
Dimension reduction is an important tool for analyzing high-dimensional data. The predictor envelope is a method of dimension reduction for regression that assumes certain linear combinations of the predictors are immaterial to the regression. The method can result in substantial gains in estimation efficiency and prediction accuracy over traditional maximum likelihood and least squares estimates. While predictor envelopes have been developed and studied for independent data, no work has been done adapting predictor envelopes to spatial data. In this work, the predictor envelope is adapted to …
This paper suggests parametrically transformed nested error regression models (TNERM), which transform the data flexibly to follow the normal linear mixed regression. We provide a procedure for estimating consistently the parameters of the proposed model and a predictor based on the consistent estimators. Then, in order to calibrate uncertainty of the transformed empirical best linear unbiased predictor, we derive prediction intervals with second-order accuracy based on the parametric bootstrap method. The proposed methods are investigated through simulation and empirical studies.
Estimating linear regression using least squares and reporting robust standard errors is very common in financial economics, and indeed, much of the social sciences and elsewhere. For thick tailed predictors under heteroskedasticity this recipe for inference performs poorly, sometimes dramatically so. Here, we develop an alternative approach which delivers an unbiased, consistent and asymptotically normal estimator so long as the means of the outcome and predictors are finite. The new method has standard errors under heteroskedasticity which are easy to reliably estimate and tests which are cl…
We consider linear regression model estimation where the covariate of interest is randomly censored. Under a non-informative censoring mechanism, one may obtain valid estimates by deleting censored observations. However, this comes at a cost of lost information and decreased efficiency, especially under heavy censoring. Other methods for dealing with censored covariates, such as ignoring censoring or replacing censored observations with a fixed number, often lead to severely biased results and are of limited practicality. Parametric methods based on maximum likelihood estimation as well as sem…
We propose a new class of estimators of the multivariate response linear regression coefficient matrix that exploits the assumption that the response and predictors have a joint multivariate Normal distribution. This allows us to indirectly estimate the regression coefficient matrix through shrinkage estimation of the parameters of the inverse regression, or the conditional distribution of the predictors given the responses. We establish a convergence rate bound for estimators in our class and we study two examples. The first example estimator exploits an assumption that the inverse regression…
We study the asymptotic behavior of a class of methods for sufficient dimension reduction in high-dimension regressions, as the sample size and number of predictors grow in various alignments. It is demonstrated that these methods are consistent in a variety of settings, particularly in abundant regressions where most predictors contribute some information on the response, and oracle rates are possible. Simulation results are presented to support the theoretical conclusion.
We develop tests of the hypothesis of no effect for selected predictors in regression, without assuming a model for the conditional distribution of the response given the predictors. Predictor effects need not be limited to the mean function and smoothing is not required. The general approach is based on sufficient dimension reduction, the idea being to replace the predictor vector with a lower-dimensional version without loss of information on the regression. Methodology using sliced inverse regression is developed in detail.
In this paper, we consider the problem of treating linear regression equation coefficients in the case of correlated predictors. It is shown that in general there are no natural ways of interpreting these coefficients similar to the case of single predictor. Nevertheless we suggest linear transformations of predictors, reducing multiple regression to a simple one and retaining the coefficient at variable of interest. The new variable can be treated as the part of the old variable that has no linear statistical dependence on other presented variables.
A variable group Y is assumed to depend upon R thematic variable groups X 1, >..., X R . We assume that components in Y depend linearly upon components in the Xr's. In this work, we propose a multiple covariance criterion which extends that of PLS regression to this multiple predictor groups situation. On this criterion, we build a PLS-type exploratory method - Structural Equation Exploratory Regression (SEER) - that allows to simultaneously perform dimension reduction in groups and investigate the linear model of the components. SEER uses the multidimensional structure of each group. An appli…
We study a dimensionality reduction technique for finite mixtures of high-dimensional multivariate response regression models. Both the dimension of the response and the number of predictors are allowed to exceed the sample size. We consider predictor selection and rank reduction to obtain lower-dimensional approximations. A class of estimators with a fast rate of convergence is introduced. We apply this result to a specific procedure, introduced in [11], where the relevant predictors are selected by the Group-Lasso.
The dual problem of testing the predictive significance of a particular covariate, and identification of the set of relevant covariates is common in applied research and methodological investigations. To study this problem in the context of functional linear regression models with predictor variables observed over a grid and a scalar response, we consider basis expansions of the functional covariates and apply the likelihood ratio test. Based on p-values from testing each predictor, we propose a new variable selection method, which is consistent in selecting the relevant predictors from set of…
Local Fréchet Regression with Riemannian Predictors
The Institute for Data Innovation in Science, Seoul National University（首尔大学数据创新科学研究所）；Department of Statistics and School of Transdisciplinary Innovations, Seoul National University（首尔大学统计系与跨学科创新学院）
This work studies finite-sample properties of the risk of the minimum-norm interpolating predictor in high-dimensional regression models. If the effective rank of the covariance matrix $Σ$ of the $p$ regression features is much larger than the sample size $n$, we show that the min-norm interpolating predictor is not desirable, as its risk approaches the risk of trivially predicting the response by 0. However, our detailed finite-sample analysis reveals, surprisingly, that this behavior is not present when the regression response and the features are {\it jointly} low-dimensional, following a w…
We consider a general nonlinear regression problem where the predictors contain measurement error. It has been recently discovered that several well-known dimension reduction methods, such as OLS, SIR and pHd, can be performed on the surrogate regression problem to produce consistent estimates for the original regression problem involving the unobserved true predictor. In this paper we establish a general invariance law between the surrogate and the original dimension reduction spaces, which implies that, at least at the population level, the two dimension reduction problems are in fact equiva…
We show that under a linearity condition on the distribution of the predictors, the coefficient in single-index regression can be estimated with the same efficiency as in the case when the link function is known. Thus, the linearity condition seems to substitute for knowing the exact conditional distribution of the response given the linear combinations of the predictors.
It is well known that individual parameters of strongly correlated predictor variables in a linear model cannot be accurately estimated by the least squares regression due to multicollinearity generated by such variables. Surprisingly, an average of these parameters can be extremely accurately estimated. We find this average and briefly discuss its applications in the least squares regression.
This paper concerns the robust regression model when the number of predictors and the number of observations grow in a similar rate. Theory for M-estimators in this regime has been recently developed by several authors [El Karoui et al., 2013, Bean et al., 2013, Donoho and Montanari, 2013]. Motivated by the inability of M-estimators to successfully estimate the Euclidean norm of the coefficient vector, we consider a Bayesian framework for this model. We suggest a two-component mixture of normals prior for the coefficients and develop a Gibbs sampler procedure for sampling from relevant posteri…
This paper studies macroeconomic forecasting and variable selection using a folded-concave penalized regression with a very large number of predictors. The penalized regression approach leads to sparse estimates of the regression coefficients, and is applicable even if the dimensionality of the model is much larger than the sample size. The first half of the paper discusses the theoretical aspects of a folded-concave penalized regression when the model exhibits time series dependence. Specifically, we show the oracle inequality and the oracle property for ultrahigh-dimensional time-dependent r…
In this paper, we propose a dimension reduction model for spatially dependent variables. Namely, we investigate an extension of the \emph{inverse regression} method under strong mixing condition. This method is based on estimation of the matrix of covariance of the expectation of the explanatory given the dependent variable, called the \emph{inverse regression}. Then, we study, under strong mixing condition, the weak and strong consistency of this estimate, using a kernel estimate of the \emph{inverse regression}. We provide the asymptotic behaviour of this estimate. A spatial predictor based …
