The term ``empirical predictor'' refers to a two-stage predictor of a linear combination of fixed and random effects. In the first stage, a predictor is obtained but it involves unknown parameters; thus, in the second stage, the unknown parameters are replaced by their estimators. In this paper, we consider mean squared errors (MSE) of empirical predictors under a general setup, where ML or REML estimators are used for the second stage. We obtain second-order approximation to the MSE as well as an estimator of the MSE correct to the same order. The general results are applied to mixed linear models to obtain a second-order approximation to the MSE of the empirical best linear unbiased predictor (EBLUP) of a linear mixed effect and an estimator of the MSE of EBLUP whose bias is correct to second order. The general mixed linear model includes the mixed ANOVA model and the longitudinal model as special cases.
A moment bound for the normalized conditional-sum-of-squares (CSS) estimate of a general autoregressive fractionally integrated moving average (ARFIMA) model with an arbitrary unknown memory parameter is derived in this paper. To achieve this goal, a uniform moment bound for the inverse of the normalized objective function is established. An important application of these results is to establish asymptotic expressions for the one-step and multi-step mean squared prediction errors (MSPE) of the CSS predictor. These asymptotic expressions not only explicitly demonstrate how the multi-step MSPE o…
This paper studies nonparametric regression with long memory (LRD) errors and predictors. First, we formulate general conditions which guarantee the standard rate of convergence for a nonparametric kernel estimator. Second, we calculate the Mean Integrated Squared Error (MISE). In particular, we show that LRD of errors may influence MISE. On the other hand, an estimator for a shape function is typically not influenced by LRD in errors. Finally, we investigate properties of a data-driven bandwidth choice. We show that Averaged Squared Error (ASE) is a good approximation of MISE, however, this i…
In regression models involving economic variables such as income, log transformation is typically taken to achieve approximate normality and stabilize the variance. However, often the interest is predicting individual values or means of the variable in the original scale. Back transformation of predicted values introduces a non-negligible bias. Moreover, assessing the uncertainty of the actual predictor is not straightforward. In this paper, a nested error model for the log transformation of the target variable is considered. Nested error models are widely used for estimation of means in subpo…
For analyzing unit-level multivariate data in small area estimation, we consider the multivariate nested error regression model (MNER) and provide the empirical best linear unbiased predictor (EBLUP) of a small area characteristic based on second-order unbiased and consistent estimators of the `within' and `between' multivariate components of variance. The second-order approximation of the mean squared error (MSE) matrix of the EBLUP and its unbiased estimator are derived in closed forms. The confidence interval with second-order accuracy is also provided analytically.
We present two approaches for linear prediction of long-memory time series. The first approach consists in truncating the Wiener-Kolmogorov predictor by restricting the observations to the last $k$ terms, which are the only available values in practice. We derive the asymptotic behaviour of the mean-squared error as $k$ tends to $ + \infty$. By contrast, the second approach is non-parametric. An AR($k$) model is fitted to the long-memory time series and we study the error that arises in this misspecified model.
We consider the problem of choosing the optimal (in the sense of mean-squared prediction error) multistep predictor for an autoregressive (AR) process of finite but unknown order. If a working AR model (which is possibly misspecified) is adopted for multistep predictions, then two competing types of multistep predictors (i.e., plug-in and direct predictors) can be obtained from this model. We provide some interesting examples to show that when both plug-in and direct predictors are considered, the optimal multistep prediction results cannot be guaranteed by correctly identifying the underlying…
We present two approaches for next step linear prediction of long memory time series. The first is based on the truncation of the Wiener-Kolmogorov predictor by restricting the observations to the last $k$ terms, which are the only available values in practice. Part of the mean squared prediction error comes from the truncation, and another part comes from the parametric estimation of the parameters of the predictor. By contrast, the second approach is non-parametric. An AR($k$) model is fitted to the long memory time series and we study the error made with this misspecified model.
This note extends a recent result of Mendelson on the supremum of a quadratic process to squared norms of functions taking values in a Banach space. Our method of proof is a reduction by a symmetrization argument and observation about the subadditivity of the generic chaining functional. We provide an application to the supremum of a linear process in the sample covariance matrix indexed by finite rank, positive definite matrices.
In this article, we first derive an explicit expression for the marginal best linear invariant predictor (BLIP) of an unobserved future order statistic based on a set of early observed ordered statistics. We then derive the joint BLIPs of two future order statistics and prove that the joint predictors are trace-efficient as well as determinant-efficient linear invariant predictors. More generally, the BLIPs are shown to possess complete mean squared predictive error matrix dominance property in the class of all linear invariant predictors of two future unobserved order statistics. Finally, the…
We study asymptotic minimax problems for estimating a $d$-dimensional regression parameter over spheres of growing dimension ($d\to \infty$). Assuming that the data follows a linear model with Gaussian predictors and errors, we show that ridge regression is asymptotically minimax and derive new closed form expressions for its asymptotic risk under squared-error loss. The asymptotic risk of ridge regression is closely related to the Stieltjes transform of the Marčenko-Pastur distribution and the spectral distribution of the predictors from the linear model. Adaptive ridge estimators are also pr…
We study conditions under which, given a dictionary $F=\{f_1,\ldots ,f_M\}$ and an i.i.d. sample $(X_i,Y_i)_{i=1}^N$, the empirical minimizer in $\operatorname {span}(F)$ relative to the squared loss, satisfies that with high probability \[R\bigl(\tilde{f}^{\mathrm{ERM}}\bigr)\leq\inf_{f\in\operatorname {span}(F)}R(f)+r_N(M),\] where $R(\cdot)$ is the squared risk and $r_N(M)$ is of the order of $M/N$. Among other results, we prove that a uniform small-ball estimate for functions in $\operatorname {span}(F)$ is enough to achieve that goal when the noise is independent of the design.
In the context of a high-dimensional linear regression model, we propose the use of an empirical correlation-adaptive prior that makes use of information in the observed predictor variable matrix to adaptively address high collinearity, determining if parameters associated with correlated predictors should be shrunk together or kept apart. Under suitable conditions, we prove that this empirical Bayes posterior concentrates around the true sparse parameter at the optimal rate asymptotically. A simplified version of a shotgun stochastic search algorithm is employed to implement the variable sele…
Volatility forecasting is crucial to risk management and portfolio construction. One particular challenge of assessing volatility forecasts is how to construct a robust proxy for the unknown true volatility. In this work, we show that the empirical loss comparison between two volatility predictors hinges on the deviation of the volatility proxy from the true volatility. We then establish non-asymptotic deviation bounds for three robust volatility proxies, two of which are based on clipped data, and the third of which is based on exponentially weighted Huber loss minimization. In particular, in…
We study the problem of predicting as well as the best linear predictor in a bounded Euclidean ball with respect to the squared loss. When only boundedness of the data generating distribution is assumed, we establish that the least squares estimator constrained to a bounded Euclidean ball does not attain the classical $O(d/n)$ excess risk rate, where $d$ is the dimension of the covariates and $n$ is the number of samples. In particular, we construct a bounded distribution such that the constrained least squares estimator incurs an excess risk of order $Ω(d^{3/2}/n)$ hence refuting a recent con…
In this paper, we consider a generalized multivariate regression problem where the responses are monotonic functions of linear transformations of predictors. We propose a semi-parametric algorithm based on the ordering of the responses which is invariant to the functional form of the transformation function. We prove that our algorithm, which maximizes the rank correlation of responses and linear transformations of predictors, is a consistent estimator of the true coefficient matrix. We also identify the rate of convergence and show that the squared estimation error decays with a rate of $o(1/…
This paper is devoted to the estimation of a partial graphical model with a structural Bayesian penalization. Precisely, we are interested in the linear regression setting where the estimation is made through the direct links between potentially high-dimensional predictors and multiple responses, since it is known that Gaussian graphical models enable to exhibit direct links only, whereas coefficients in linear regressions contain both direct and indirect relations (due \textit{e.g.} to strong correlations among the variables). A smooth penalty reflecting a generalized Gaussian Bayesian prior …
Linear wavelet density estimators are wavelet projections of the empirical measure based on independent, identically distributed observations. We study here the law of the iterated logarithm (LIL) and a Berry-Esseen type theorem. These results are proved under different assumptions on the density $f$ that are different from those needed for similar results in the case of convolution kernels (KDE): whereas the smoothness requirements are much less stringent than for the KDE, Riemann integrability assumptions are needed in order to compute the asymptotic variance, which gives the scaling constan…
INRIA - Département d’Informatique de l’École Normale Supérieure（法国国家信息与自动化研究所 - 巴黎高等师范学院计算机系）；PSL Research University（巴黎文理研究大学）
We consider learning methods based on the regularization of a convex empirical risk by a squared Hilbertian norm, a setting that includes linear predictors and non-linear predictors through positive-definite kernels. In order to go beyond the generic analysis leading to convergence rates of the excess risk as $O(1/\sqrt{n})$ from $n$ observations, we assume that the individual losses are self-concordant, that is, their third-order derivatives are bounded by their second-order derivatives. This setting includes least-squares, as well as all generalized linear models such as logistic and softmax…
Bagging can significantly improve the generalization performance of unstable machine learning algorithms such as trees or neural networks. Though bagging is now widely used in practice and many empirical studies have explored its behavior, we still know little about the theoretical properties of bagged predictions. In this paper, we theoretically investigate how the bagging method can reduce the Mean Squared Error (MSE) when applied on a statistical estimator. First, we prove that for any estimator, increasing the number of bagged estimators $N$ in the average can only reduce the MSE. This int…
