Communication-Efficient Distributed Estimator for Generalized Linear Models with a Diverging Number of Covariates
School of Applied Sciences, Beijing Information Science and Technology University（北京信息科技大学应用科学学院）；School of Statistics, Renmin University of China（中国人民大学统计学院）；School of Statistics and Mathematics, Central University of Finance and Economics（中央财经大学统计与数学学院）
Ping Zhou, Zhen Yu, Jingyi Ma, Maozai Tian, Ye Fan
Distributed statistical inference has recently attracted immense attention. The asymptotic efficiency of the maximum likelihood estimator (MLE), the one-step MLE, and the aggregated estimating equation estimator are established for generalized linear models under the "large $n$, diverging $p_n$" framework, where the dimension of the covariates $p_n$ grows to infinity at a polynomial rate $o(n^α)$ for some $0<α<1$. Then a novel method is proposed to obtain an asymptotically efficient estimator for large-scale distributed data by two rounds of communication. In this novel method, the assumption on the number of servers is more relaxed and thus practical for real-world applications. Simulations and a case study demonstrate the satisfactory finite-sample performance of the proposed estimators.
For statistical inference on regression models with a diverging number of covariates, the existing literature typically makes sparsity assumptions on the inverse of the Fisher information matrix. Such assumptions, however, are often violated under Cox proportion hazards models, leading to biased estimates with under-coverage confidence intervals. We propose a modified debiased lasso approach, which solves a series of quadratic programming problems to approximate the inverse information matrix without posing sparse matrix assumptions. We establish asymptotic results for the estimated regression…
Clustered binary data with a large number of covariates have become increasingly common in many scientific disciplines. This paper develops an asymptotic theory for generalized estimating equations (GEE) analysis of clustered binary data when the number of covariates grows to infinity with the number of clusters. In this "large $n$, diverging $p$" framework, we provide appropriate regularity conditions and establish the existence, consistency and asymptotic normality of the GEE estimator. Furthermore, we prove that the sandwich variance formula remains valid. Even when the working correlation …
Modeling and drawing inference on the joint associations between single nucleotide polymorphisms and a disease has sparked interest in genome-wide associations studies. In the motivating Boston Lung Cancer Survival Cohort (BLCSC) data, the presence of a large number of single nucleotide polymorphisms of interest, though smaller than the sample size, challenges inference on their joint associations with the disease outcome. In similar settings, we find that neither the de-biased lasso approach (van de Geer et al. 2014), which assumes sparsity on the inverse information matrix, nor the standard …
We propose generalized additive partial linear models for complex data which allow one to capture nonlinear patterns of some covariates, in the presence of linear components. The proposed method improves estimation efficiency and increases statistical power for correlated data through incorporating the correlation information. A unique feature of the proposed method is its capability of handling model selection in cases where it is difficult to specify the likelihood function. We derive the quadratic inference function-based estimators for the linear coefficients and the nonparametric function…
This short note is to point the reader to notice that the proof of high dimensional asymptotic normality of MLE estimator for logistic regression under the regime $p_n=o(n)$ given in paper: "Maximum likelihood estimation in logistic regression models with a diverging number of covariates. Electronic Journal of Statistics, 6, 1838-1846." is wrong.
We study problems with multiple missing covariates and partially observed responses. We develop a new framework to handle complex missing covariate scenarios via inverse probability weighting, regression adjustment, and a multiply-robust procedure. We apply our framework to three classical problems: the Cox model from survival analysis, missing response, and binary treatment from causal inference. We also discuss how to handle missing covariates in these scenarios, and develop associated identifying theories and asymptotic theories. We apply our procedure to simulations and an Alzheimer's dise…
In this paper, we propose an empirical likelihood-based weighted estimator of regression parameter in quantile regression model with nonignorable missing covariates. The proposed estimator is computationally simple and achieves semiparametric efficiency if the probability of missingness on the fully observed variables is correctly specified. The efficiency gain of the proposed estimator over the complete-case-analysis estimator is quantified theoretically and illustrated via simulation and a real data application.
Evaluating the joint significance of covariates is of fundamental importance in a wide range of applications. To this end, p-values are frequently employed and produced by algorithms that are powered by classical large-sample asymptotic theory. It is well known that the conventional p-values in Gaussian linear model are valid even when the dimensionality is a non-vanishing fraction of the sample size, but can break down when the design matrix becomes singular in higher dimensions or when the error distribution deviates from Gaussianity. A natural question is when the conventional p-values in g…
The classic integrated conditional moment test is a promising method for testing regression model misspecification. However, it severely suffers from the curse of dimensionality. To extend it to handle the testing problem for parametric multi-index models with diverging number of covariates, we investigate three issues in inference in this paper. First, we study the consistency and asymptotically linear representation of the least squares estimator of the parameter matrix at faster rates of divergence than those in the literature for nonlinear models. Second, we propose, via sufficient dimensi…
We consider estimating the density of a response conditioning on an error-prone covariate. Motivated by two existing kernel density estimators in the absence of covariate measurement error, we propose a method to correct the existing estimators for measurement error. Asymptotic properties of the resultant estimators under different types of measurement error distributions are derived. Moreover, we adjust bandwidths readily available from existing bandwidth selection methods developed for error-free data to obtain bandwidths for the new estimators. Extensive simulation studies are carried out t…
Adaptive deep nonparametric regression from dependent data under covariate shift
Université Jean Monnet, ICJ UMR5208, CNRS, Ecole Centrale de Lyon, INSA Lyon, Universite Claude Bernard Lyon 1（里昂 Jean Monnet 大学，ICJ UMR5208，国家科学研究中心，里昂中央理工大学，里昂国家理工学校，里昂大学 Claude Bernard 1）
In real supervised learning scenarios, it is not uncommon that the training and test sample follow different probability distributions, thus rendering the necessity to correct the sampling bias. Focusing on a particular covariate shift problem, we derive high probability confidence bounds for the kernel mean matching (KMM) estimator, whose convergence rate turns out to depend on some regularity measure of the regression function and also on some capacity measure of the kernel. By comparing KMM with the natural plug-in estimator, we establish the superiority of the former hence provide concrete…
The aim of this article is to propose a novel kernel estimator of the baseline function in a general high-dimensional Cox model, for which we derive non-asymptotic rates of convergence. To construct our estimator, we first estimate the regression parameter in the Cox model via a Lasso procedure. We then plug this estimator into the classical kernel estimator of the baseline function, obtained by smoothing the so-called Breslow estimator of the cumulative baseline function. We propose and study an adaptive procedure for selecting the bandwidth, in the spirit of Gold-enshluger and Lepski (2011).…
In this paper we discuss improved estimators for the regression and the dispersion parameters in an extended class of dispersion models (Jørgensen, 1996). This class extends the regular dispersion models by letting the dispersion parameter vary throughout the observations, and contains the dispersion models as particular case. General formulae for the second-order bias are obtained explicitly in dispersion models with dispersion covariates, which generalize previous results by Botter and Cordeiro (1998), Cordeiro and McCullagh (1991), Cordeiro and Vasconcellos (1999), and Paula (1992). The pra…
In real applications of the linear model, the explanatory variables are very often naturally grouped, the most common example being the multivariate variance analysis. In the present paper, a quantile model with structure group is considered, the number of groups can diverge with sample size. We introduce and study the adaptive elastic-net group estimator, for improving the parameter estimation accuracy. This method allows automatic selection, with a probability converging to one, of significant groups and further the non zero parameter estimators are asymptotically normal. The convergence rat…
We study the asymptotic properties of bridge estimators in sparse, high-dimensional, linear regression models when the number of covariates may increase to infinity with the sample size. We are particularly interested in the use of bridge estimators to distinguish between covariates whose coefficients are zero and covariates whose coefficients are nonzero. We show that under appropriate conditions, bridge estimators correctly select covariates with nonzero coefficients with probability converging to one and that the estimators of nonzero coefficients have the same asymptotic distribution that …
An important phenomenon in high dimensional biological data is the presence of unobserved covariates that can have a significant impact on the measured response. When these factors are also correlated with the covariate(s) of interest (i.e. disease status), ignoring them can lead to increased type I error and spurious false discovery rate estimates. We show that depending on the strength of this correlation and the informativeness of the observed data for the latent factors, previously proposed estimators for the effect of the covariate of interest that attempt to account for unobserved covari…
In this paper, we consider a weighted local linear estimator based on the inverse selection probability for nonparametric regression with missing covariates at random. The asymptotic distribution of the maximal deviation between the estimator and the true regression function is derived and an asymptotically accurate simultaneous confidence band is constructed. The estimator for the regression function is shown to be oracally efficient in the sense that it is uniformly indistinguishable from that when the selection probabilities are known. Finite sample performance is examined via simulation st…
Since survival data occur over time, often important covariates that we wish to consider also change over time. Such covariates are referred as time-dependent covariates. Quantile regression offers flexible modeling of survival data by allowing the covariates to vary with quantiles. This paper provides a novel quantile regression model accommodating time-dependent covariates, for analyzing survival data subject to right censoring. Our simple estimation technique assumes the existence of instrumental variables. In addition, we present a doubly-robust estimator in the sense of Robins and Rotnitz…
We propose a new class of semiparametric regression models of mean residual life for censored outcome data. The models, which enable us to estimate the expected remaining survival time and generalize commonly used mean residual life models, also conduct covariate dimension reduction. Using the geometric approaches in semiparametrics literature and the martingale properties with survival data, we propose a flexible inference procedure that relaxes the parametric assumptions on the dependence of mean residual life on covariates and how long a patient has lived. We show that the estimators for th…
Randomized block factorial experiments are widely used in industrial engineering, clinical trials, and social science. Researchers often use a linear model and analysis of covariance to analyze experimental results; however, limited studies have addressed the validity and robustness of the resulting inferences because assumptions for a linear model might not be justified by randomization in randomized block factorial experiments. In this paper, we establish a new finite population joint central limit theorem for usual (unadjusted) factorial effect estimators in randomized block $2^K$ factorial…
In this paper, a nonparametric estimator is proposed for estimating the L1-median for multivariate conditional distribution when the covariates take values in an infinite dimensional space. The multivariate case is more appropriate to predict the components of a vector of random variables simultaneously rather than predicting each of them separately. While estimating the conditional L1-median function using the well-known Nadarya-Waston estimator, we establish the strong consistency of this estimator as well as the asymptotic normality. We also present some simulations and provide how to built…
We consider nonparametric estimation of a regression curve when the data are observed with multiplicative distortion which depends on an observed confounding variable. We suggest several estimators, ranging from a relatively simple one that relies on restrictive assumptions usually made in the literature, to a sophisticated piecewise approach that involves reconstructing a smooth curve from an estimator of a constant multiple of its absolute value, and which can be applied in much more general scenarios. We show that, although our nonparametric estimators are constructed from predictors of the…
Contamination of covariates by measurement error is a classical problem in multivariate regression, where it is well known that failing to account for this contamination can result in substantial bias in the parameter estimators. The nature and degree of this effect on statistical inference is also understood to crucially depend on the specific distributional properties of the measurement error in question. When dealing with functional covariates, measurement error has thus far been modelled as additive white noise over the observation grid. Such a setting implicitly assumes that the error ari…
Functional covariates are common in many medical, biodemographic, and neuroimaging studies. The aim of this paper is to study functional Cox models with right-censored data in the presence of both functional and scalar covariates. We study the asymptotic properties of the maximum partial likelihood estimator and establish the asymptotic normality and efficiency of the estimator of the finite-dimensional estimator. Under the framework of reproducing kernel Hilbert space, the estimator of the coefficient function for a functional covariate achieves the minimax optimal rate of convergence under a…
Transportable inference using target population summary statistics under covariate shift
Academy of Mathematics and Systems Science Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Department of Biostatistics Mailman School of Public Health Columbia University（哥伦比亚大学麦尔曼公共卫生学院生物统计学系）；Department of Epidemiology & Biostatistics University of California at San Francisco（旧金山大学生物统计与流行病学系）
The relationship between a time-dependent covariate and survival times is usually evaluated via the Cox model. Time-dependent covariates are generally available as longitudinal data collected regularly during the course of the study. A frequent problem, however, is the occurence of missing covariate data. A recent approach to estimation in the Cox model in this case jointly models survival and the longitudinal covariate. However, theoretical justification of this approach is still lacking. In this paper we prove existence and consistency of the maximum likelihood estimators in a joint model. T…
The generalized varying coefficient partially linear model with growing number of predictors arises in many contemporary scientific endeavor. In this paper we set foot on both theoretical and practical sides of profile likelihood estimation and inference. When the number of parameters grows with sample size, the existence and asymptotic normality of the profile likelihood estimator are established under some regularity conditions. Profile likelihood ratio inference for the growing number of parameters is proposed and Wilk's phenomenon is demonstrated. A new algorithm, called the accelerated pr…
We analyze the statistical properties of nonparametric regression estimators using covariates which are not directly observable, but have be estimated from data in a preliminary step. These so-called generated covariates appear in numerous applications, including two-stage nonparametric regression, estimation of simultaneous equation models or censored regression models. Yet so far there seems to be no general theory for their impact on the final estimator's statistical properties. Our paper provides such results. We derive a stochastic expansion that characterizes the influence of the generat…
This paper deals with a linear model of regression on quantiles when the explanatory variable takes values in some functional space and the response is scalar. We propose a spline estimator of the functional coefficient that minimizes a penalized L1 type criterion. Then, we study the asymptotic behavior of this estimator. The penalization is of primary importance to get existence and convergence.
Recent research has focused on $\ell_1$ penalized least squares (Lasso) estimators for high-dimensional linear regressions in which the number of covariates $p$ is considerably larger than the sample size $n$. However, few studies have examined the properties of the estimators when the errors and/or the covariates are serially dependent. In this study, we investigate the theoretical properties of the Lasso estimator for a linear regression with a random design and weak sparsity under serially dependent and/or nonsubGaussian errors and covariates. In contrast to the traditional case, in which t…
We consider inference in linear regression models that is robust to heteroskedasticity and the presence of many control variables. When the number of control variables increases at the same rate as the sample size the usual heteroskedasticity-robust estimators of the covariance matrix are inconsistent. Hence, tests based on these estimators are size distorted even in large samples. An alternative covariance-matrix estimator for such a setting is presented that complements recent work by Cattaneo, Jansson and Newey (2018). We provide high-level conditions for our approach to deliver (asymptotic…
This paper considers maximum likelihood (ML) estimation in a large class of models with hidden Markov regimes. We investigate consistency of the ML estimator and local asymptotic normality for the models under general conditions which allow for autoregressive dynamics in the observable process, Markov regime sequences with covariate-dependent transition matrices, and possible model misspecification. A Monte Carlo study examines the finite-sample properties of the ML estimator in correctly specified and misspecified models. An empirical application is also discussed.
Motivated by normalizing DNA microarray data and by predicting the interest rates, we explore nonparametric estimation of additive models with highly correlated covariates. We introduce two novel approaches for estimating the additive components, integration estimation and pooled backfitting estimation. The former is designed for highly correlated covariates, and the latter is useful for nonhighly correlated covariates. Asymptotic normalities of the proposed estimators are established. Simulations are conducted to demonstrate finite sample behaviors of the proposed estimators, and real data ex…
We consider estimation in a particular semiparametric regression model for the mean of a counting process with ``panel count'' data. The basic model assumption is that the conditional mean function of the counting process is of the form $E\{\mathbb{N}(t)|Z\}=\exp(β_0^TZ)Λ_0(t)$ where $Z$ is a vector of covariates and $Λ_0$ is the baseline mean function. The ``panel count'' observation scheme involves observation of the counting process $\mathbb{N}$ for an individual at a random number $K$ of random time points; both the number and the locations of these time points may differ across individual…
Although complete randomization ensures covariate balance on average, the chance for observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance criterion, generally resulting in better covariate balance and more precise estimates of causal effects. Previous theory has derived finite sample theory for rerandomization under the assumptions of equal treatment group sizes, Gaussian covariate and outcome distributions, or additive causal effects,…
In this paper we consider the estimation of population size from one-source capture--recapture data, that is, a list in which individuals can potentially be found repeatedly and where the question is how many individuals are missed by the list. As a typical example, we provide data from a drug user study in Bangkok from 2001 where the list consists of drug users who repeatedly contact treatment institutions. Drug users with 1, 2, 3$,...$ contacts occur, but drug users with zero contacts are not present, requiring the size of this group to be estimated. Statistically, these data can be consider…
Consider a high-dimensional linear regression problem, where the number of covariates is larger than the number of observations and the interest is in estimating the conditional variance of the response variable given the covariates. A conditional and unconditioned framework are considered, where conditioning is with respect to the covariates, which are ancillary to the parameter of interest. In recent papers, a consistent estimator was developed in the unconditional framework when the marginal distribution of the covariates is normal with known mean and variance. In the present work, a certai…
Feature selection procedures for spatial point processes parametric intensity estimation have been recently developed since more and more applications involve a large number of covariates. In this paper, we investigate the setting where the number of covariates diverges as the domain of observation increases. In particular, we consider estimating equations based on Campbell theorems derived from Poisson and logistic regression likelihoods regularized by a general penalty function. We prove that, under some conditions, the consistency, the sparsity, and the asymptotic normality are valid for su…
Completely randomized experiments have been the gold standard for drawing causal inference because they can balance all potential confounding on average. However, they may suffer from unbalanced covariates for realized treatment assignments. Rerandomization, a design that rerandomizes the treatment assignment until a prespecified covariate balance criterion is met, has recently got attention due to its easy implementation, improved covariate balance and more efficient inference. Researchers have then suggested to use the treatment assignments that minimize the covariate imbalance, namely the o…
