The lasso has been studied extensively as a tool for estimating the coefficient vector in the high-dimensional linear model; however, considerably less is known about estimating the error variance in this context. In this paper, we propose the natural lasso estimator for the error variance, which maximizes a penalized likelihood objective. A key aspect of the natural lasso is that the likelihood is expressed in terms of the natural parameterization of the multiparameter exponential family of a Gaussian with unknown mean and variance. The result is a remarkably simple estimator of the error variance with provably good performance in terms of mean squared error. These theoretical results do not require placing any assumptions on the design matrix or the true regression coefficients. We also propose a companion estimator, called the organic lasso, which theoretically does not require tuning of the regularization parameter. Both estimators do well empirically compared to preexisting methods, especially in settings where successful recovery of the true support of the coefficient vector is hard. Finally, we show that existing methods can do well under fewer assumptions than previously known, thus providing a fuller story about the problem of estimating the error variance in high-dimensional linear models.
This paper investigates the large sample properties of the variance, weights, and risk of high-dimensional portfolios where the inverse of the covariance matrix of excess asset returns is estimated using a technique called nodewise regression. Nodewise regression provides a direct estimator for the inverse covariance matrix using the Least Absolute Shrinkage and Selection Operator (Lasso) of Tibshirani (1994) to estimate the entries of a sparse precision matrix. We show that the variance, weights, and risk of the global minimum variance portfolios and the Markowitz mean-variance portfolios are…
National Dong Hwa University（国立东华大学）；Seoul National University（首尔国立大学）；Michigan State University（密歇根州立大学）；University of Maryland, Baltimore（马里兰大学巴尔的摩分校）；Sumaq Life LLC（Sumaq Life有限责任公司）；Yonsei University（延世大学）
In system identification, estimating parameters of a model using limited observations results in poor identifiability. To cope with this issue, we propose a new method to simultaneously select and estimate sensitive parameters as key model parameters and fix the remaining parameters to a set of typical values. Our method is formulated as a nonlinear least squares estimator with L1-regularization on the deviation of parameters from a set of typical values. First, we provide consistency and oracle properties of the proposed estimator as a theoretical foundation. Second, we provide a novel approa…
Mean and variance estimation in high-dimensional heteroscedastic models with non-convex penalties
尽管异方差（非常数样本方差）在统计数据集中很普遍，但在高维统计文献中，它在很大程度上被忽略了。最近，研究表明，Lasso可以适应异方差误差，只需进行微小的算法修改（Belloni等人，2012; Gautier and Tsybakov，2013）.本文研究了高维稀疏参数下的线性均值模型和对数线性方差模型的异方差回归问题。在这项工作中，我们提出了估计方差后Lasso时尚，其次是加权最小二乘均值估计。这些步骤使用了Fan和Li（2001）中的非凸罚分，这使我们能够证明后Lasso方差和均值参数估计的预言性质。我们用实验来加强我们的理论发现。
We develop efficient binary (i.e., 1-bit) and multi-bit coding schemes for estimating the scale parameter of $α$-stable distributions. The work is motivated by the recent work on one scan 1-bit compressed sensing (sparse signal recovery) using $α$-stable random projections, which requires estimating of the scale parameter at bits-level. Our technique can be naturally applied to data stream computations for estimating the $α$-th frequency moment. In fact, the method applies to the general scale family of distributions, not limited to $α$-stable distributions. Due to the heavy-tailed nature of $…
This paper introduces a general framework for estimating variance components in the linear mixed models via general unbiased estimating equations, which include some well-used estimators such as the restricted maximum likelihood estimator. We derive the asymptotic covariance matrices and second-order biases under general estimating equations without assuming the normality of the underlying distributions and identify a class of second-order unbiased estimators of variance components. It is also shown that the asymptotic covariance matrices and second-order biases do not depend on whether the re…
Missing data is a common challenge in biomedical research. This fact, along with growing dataset volumes of the modern era, make the issue of computationally-efficient analysis with missing data of crucial practical importance. A general computationally-efficient estimation framework for dealing with missing data is the pseudo-expected estimating equations (PEEE) approach. The method is applicable with any parametric model for which estimation involves the solution of a set of estimating equations, such as likelihood score equations. A key limitation of the PEEE approach is that there is curre…
Estimation of a treatment effect by a regression discontinuity design faces a severe challenge when the running variable contains measurement errors since the errors smoothen the discontinuity on which the identification depends. The existing studies show that the variance of the measurement errors plays a vital role in both bias correction and identification under such situations. However, the methodologies to estimate the variance from data are relatively undeveloped. This paper proposes two estimators for the variance of measurement errors of running variables of sharp regression continuity…
This paper provides a design-based framework for variance (bound) estimation in experimental analysis. Results are applicable to virtually any combination of experimental design, linear estimator (e.g., difference-in-means, OLS, WLS) and variance bound, allowing for unified treatment and a basis for systematic study and comparison of designs using matrix spectral analysis. A proposed variance estimator reproduces Eicker-Huber-White (aka. "robust", "heteroskedastic consistent", "sandwich", "White", "Huber-White", "HC", etc.) standard errors and "cluster-robust" standard errors as special cases.…
In this paper, I develop a formula for estimating Bayes factors directly from minimal summary statistics produced in repeated measures analysis of variance designs. The formula, which requires knowing only the $F$-statistic, the number of subjects, and the number of repeated measurements per subject, is based on the BIC approximation of the Bayes factor, a common default method for Bayesian computation with linear models. In addition to providing computational examples, I report a simulation study in which I demonstrate that the formula compares favorably to a recently developed, more complex …
We often estimate a parameter of interest psi when the identifying conditions involve a nuisance parameter theta. Examples from causal inference are Inverse Probability Weighting, Marginal Structural Models and Structural Nested Models. To estimate treatment effects from observational data, these methods posit a (pooled) logistic regression model for the treatment and/or censoring probabilities and estimate these first. These methods are all based on unbiased estimating equations. First, we provide a general formula for the variance of the parameter of interest psi when the nuisance parameter …
Estimating with kernel smoothers the mean of functional data in a finite population setting. A note on variance estimation in presence of partially observed trajectories
This paper provides a framework for estimating the mean and variance of a high-dimensional normal density. The main setting considered is a fixed number of vector following a high-dimensional normal distribution with unknown mean and diagonal covariance matrix. The diagonal covariance matrix can be known or unknown. If the covariance matrix is unknown, the sample size can be as small as $2$. The proposed estimator is based on the idea that the unobserved pairs of mean and variance for each dimension are drawn from an unknown bivariate distribution, which we model as a mixture of normal-inverse…
The development of a complex disease is an intricate interplay of genetic and environmental factors. "Heritability" is defined as the proportion of total trait variance due to genetic factors within a given population. Studies with monozygotic (MZ) and dizygotic (DZ) twins allow us to estimate heritability by fitting an "ACE" model which estimates the proportion of trait variance explained by additive genetic (A), common shared environment (C), and unique non-shared environmental (E) latent effects, thus helping us better understand disease risk and etiology. In this paper, we develop a flexib…
In this article, we develop methods for sample size and power calculations in four-level intervention studies when intervention assignment is carried out at any level, with a particular focus on cluster randomized trials (CRTs). CRTs involving four levels are becoming popular in health care research, where the effects are measured, for example, from evaluations (level 1) within participants (level 2) in divisions (level 3) that are nested in clusters (level 4). In such multi-level CRTs, we consider three types of intraclass correlations between different evaluations to account for such cluster…
Random-effects meta-analysis requires an estimate of the between-study variance, $τ^2$. We study methods of estimation of $τ^2$ and its confidence interval in meta-analysis of odds ratio, and also the performance of related estimators of the overall effect. We provide results of extensive simulations on five point estimators of $τ^2$ (the popular methods of DerSimonian-Laird, restricted maximum likelihood, and Mandel and Paule; the less-familiar method of Jackson; and the new method (KD) based on the improved approximation to the distribution of the Q statistic by Kulinskaya and Dollinger (201…
We propose a two-stage estimation method of variance components in time series models known as FDSLRMs, whose observations can be described by a linear mixed model (LMM). We based estimating variances, fundamental quantities in a time series forecasting approach called kriging, on the empirical (plug-in) best linear unbiased predictions of unobservable random components in FDSLRM. The method, providing invariant non-negative quadratic estimators, can be used for any absolutely continuous probability distribution of time series data. As a result of applying the convex optimization and the LMM m…
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure standardized mean difference (SMD), we provide the results from extensive simulations on five point estimators of $τ^2$ (the popular methods of DerSimonian-Laird, restricted maximum likelihood, and Mandel and Paule (MP); the less-familiar method of Jackson; the…
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure mean difference (MD), we review five point estimators of $τ^2$ (the popular methods of DerSimonian-Laird, restricted maximum likelihood, and Mandel and Paule (MP); the less-familiar method of Jackson; and a new method (WT) based on the improved approximation to…
Gene expression is stochastic and displays variation ("noise") both within and between cells. Intracellular (intrinsic) variance can be distinguished from extracellular (extrinsic) variance by applying the law of total variance to data from two-reporter assays that probe expression of identical gene pairs in single-cells. We examine established formulas for the estimation of intrinsic and extrinsic noise and provide interpretations of them in terms of a hierarchical model. This allows us to derive corrections that minimize the mean squared error, an objective that may be important when sample …
In this paper, we propose a modified Newton-Raphson algorithm to estimate the frequency parameter in the fundamental frequency model in presence of an additive stationary error. The proposed estimator is super efficient in nature in the sense that its asymptotic variance is less than the asymptotic variance of the least squares estimator. With a proper step factor modification, the proposed modified Newton-Raphson algorithm produces an estimator with the rate $O_p(n^{-\frac{3}{2}})$, the same rate as the least squares estimator. Numerical experiments are performed for different sample sizes, d…
Consider one realization of a continuous-time Gaussian process $Z$ which belongs to the Mat\' ern family with known ``regularity'' index $ν>0$. For estimating the autocorrelation-range and the variance of $Z$ from $n$ observations on a fine grid, we studied in Girard (2016) the GE-EV method which simply retains the empirical variance (EV) and equates it to a candidate ``Gibbs energy (GE)'' i.e.~the quadratic form ${\bf z}^T R^{-1} {\bf z}/n$ where ${\bf z}$ is the vector of observations and $R$ is the autocorrelation matrix for ${\bf z}$ associated with a candidate range. The present study con…
Methods for random-effects meta-analysis require an estimate of the between-study variance, $τ^2$. The performance of estimators of $τ^2$ (measured by bias and coverage) affects their usefulness in assessing heterogeneity of study-level effects, and also the performance of related estimators of the overall effect. For the effect measure log-response-ratio (LRR, also known as the logarithm of the ratio of means, RoM), we review four point estimators of $τ^2$ (the popular methods of DerSimonian-Laird (DL), restricted maximum likelihood, and Mandel and Paule (MP), and the less-familiar method of …
In the analysis of observational studies, inverse probability weighting (IPW) is commonly used to consistently estimate the average treatment effect (ATE) or the average treatment effect in the treated (ATT). The variance of the IPW ATE estimator is often estimated by assuming the weights are known and then using the so-called "robust" (Huber-White) sandwich estimator, which results in conservative standard error (SE) estimation. Here it is shown that using such an approach when estimating the variance of the IPW ATT estimator does not necessarily result in conservative SE estimates. That is, …
Finite-sample bias-variance tradeoff with variables related to trial participation inserted into causal forest models for ensuring generalizability
Division of Preventive Medicine, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School（预防医学系，内科，布莱根妇女医院和哈佛医学院）；Department of Epidemiology, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University（流行病学科，医药齿学研究院，冈山大学）；Harris School of Public Policy, University of Chicago（哈里斯公共政策学院，芝加哥大学）
The nested error regression model is a useful tool for analyzing clustered (grouped) data, and is especially used in small area estimation. The classical nested error regression model assumes normality of random effects and error terms, and homoscedastic variances. However, these assumptions are often violated in real applications and more flexible models are required. This article proposes a nested error regression model with heteroscedastic variances, where the normality for the underlying distributions is not assumed. We propose the structure of heteroscedastic variances by using some speci…
In this paper, a practical estimation method for a regression model is proposed using semiparametric efficient score functions applicable to data with various shapes of errors. First, I derive semiparametric efficient score vectors for a homoscedastic regression model without any assumptions of errors. Next, the semiparametric efficient score function can be modified assuming a certain parametric distribution of errors according to the shape of the error distribution or by estimating the error distribution non-parametrically. Nonparametric methods for errors can be used to estimate the paramet…
A new method for estimating structural equation models (SEM) is proposed and evaluated. In contrast to most other methods, it is based directly on the data, not on the covariance matrix of the data. The new approach is flexible enough to handle non-linear and non-smooth models and allows to model various constraints. Principle strengths and weaknesses of this approach are discussed and simulation studies are performed to reveal problems and potentials of this approach.
The inverse probability weighting approach is popular for evaluating treatment effects in observational studies, but extreme propensity scores could bias the estimator and induce excessive variance. Recently, the overlap weighting approach has been proposed to alleviate this problem, which smoothly down-weighs the subjects with extreme propensity scores. Although advantages of overlap weighting have been extensively demonstrated in literature with continuous and binary outcomes, research on its performance with time-to-event or survival outcomes is limited. In this article, we propose two weig…
Motivated by inferring cellular signaling networks using noisy flow cytometry data, we develop procedures to draw inference for Bayesian networks based on error-prone data. Two methods for inferring causal relationships between nodes in a network are proposed based on penalized estimation methods that account for measurement error and encourage sparsity. We discuss consistency of the proposed network estimators and develop an approach for selecting the tuning parameter in the penalized estimation methods. Empirical studies are carried out to compare the proposed methods and a naive method that…
We present a general M-estimation framework for inference on the wavelet variance. This framework generalizes the results on the scale-wise properties of the standard estimator and extends them to deliver the joint asymptotic properties of the estimated wavelet variance vector. Moreover, this is achieved by extending the estimation of the wavelet variance to multidimensional random fields and by stating the necessary conditions for these properties to hold when the size of the wavelet variance vector goes to infinity with the sample size. Finally, these results generally hold when using bounde…
Estimating prevalence, the fraction of a population with a certain medical condition, is fundamental to epidemiology. Traditional methods rely on classification of test samples taken at random from a population. Such approaches to estimating prevalence are biased and have uncontrolled uncertainty. Here, we construct a new, unbiased, minimum variance estimator for prevalence. Recent result show that prevalence can be estimated from counting arguments that compare the fraction of samples in an arbitrary subset of the measurement space to what is expected from conditional probability models of th…
Reference based multiple imputation methods have become popular for handling missing data in randomised clinical trials. Rubin's variance estimator is well known to be biased compared to the reference based imputation estimator's true repeated sampling variance. Somewhat surprisingly given the increasingly popularity of these methods, there has been relatively little debate in the literature as to whether Rubin's variance estimator or alternative (smaller) variance estimators targeting the repeated sampling variance are more appropriate. We review the arguments made on both sides of this debat…
Spatial microeconometric studies typically suffer from various forms of inaccuracies that are not present when dealing with the classical regional spatial econometrics models. Among those, missing data, locational errors, sampling without a formal sample design, measurement errors and misalignment are the typical sources of inaccuracy that can affects the results in a spatial microeconometric analysis. In this paper, we have examined the effects of measurement error introduced in a logistic model by random geo-masking, when distances are used as predictors. Extending the classical results on t…
In this paper we study simulation-based methods for estimating gradients in stochastic networks. We derive a new method of calculating weak derivative estimator using importance sampling transform, and our method has less computational cost than the classical method. In the context of M/M/1 queueing network and stochastic activity network, we analytically show that our new method won't result in a great increase of sample variance of the estimators. Our numerical experiments show that under same simulation time, the new method can yield a narrower confidence interval of the true gradient than …
Lag windows are commonly used in time series, econometrics, steady-state simulation, and Markov chain Monte Carlo to estimate time-average covariance matrices. In the presence of positive correlation of the underlying process, estimators of this matrix almost always exhibit significant negative bias, leading to undesirable finite-sample properties. We propose a new family of lag windows specifically designed to improve finite-sample performance by offsetting this negative bias. Any existing lag window can be adapted into a lugsail equivalent with no additional assumptions. We use these lag win…
