In this paper, we propose new semiparametric procedures for making inference on linear functionals and their functions of two semicontinuous populations. The distribution of each population is usually characterized by a mixture of a discrete point mass at zero and a continuous skewed positive component, and hence such distribution is semicontinuous in the nature. To utilize the information from both populations, we model the positive components of the two mixture distributions via a semiparametric density ratio model. Under this model setup, we construct the maximum empirical likelihood estimators of the linear functionals and their functions, and establish the asymptotic normality of the proposed estimators. We show the proposed estimators of the linear functionals are more efficient than the fully nonparametric ones. The developed asymptotic results enable us to construct confidence regions and perform hypothesis tests for the linear functionals and their functions. We further apply these results to several important summary quantities such as the moments, the mean ratio, the coefficient of variation, and the generalized entropy class of inequality measures. Simulation studies demonstrate the advantages of our proposed semiparametric method over some existing methods. Two real data examples are provided for illustration.
The Gini index is a popular inequality measure with many applications in social and economic studies. This paper studies semiparametric inference on the Gini indices of two semicontinuous populations. We characterize the distribution of each semicontinuous population by a mixture of a discrete point mass at zero and a continuous skewed positive component. A semiparametric density ratio model is then employed to link the positive components of the two distributions. We propose the maximum empirical likelihood estimators of the two Gini indices and their difference, and further investigate the a…
This paper considers a wide family of semiparametric repeated measures regression models, in which the main interest is on estimating population-level quantities such as mean, variance, probabilities etc. Examples of our framework include generalized linear models for clustered/longitudinal data, among many others. We derive plug-in kernel-based estimators of the population level quantities and derive their asymptotic distribution. An example involving estimation of the survival function of hemoglobin measures in the Kenya hemoglobin study data is presented to demonstrate our methodology.
When the target variable exhibits a semicontinuous behaviour (i.e. a point mass in a single value and a continuous distribution elsewhere) parametric `two-part regression models' have been extensively used and investigated. In this paper, a semiparametric Bayesian two-part regression model for dealing with such variables is proposed. The model allows a semiparametric expression for the two part of the model by using Dirichlet processes. A motivating example (in the `small area estimation' framework) based on pseudo-real data on grapewine production in Tuscany, is used to evaluate the capabilit…
In this paper, we consider nonparametric multidimensional finite mixture models and we are interested in the semiparametric estimation of the population weights. Here, the i.i.d. observations are assumed to have at least three components which are independent given the population. We approximate the semiparametric model by projecting the conditional distributions on step functions associated to some partition. Our first main result is that if we refine the partition slowly enough, the associated sequence of maximum likelihood estimators of the weights is asymptotically efficient, and the poste…
We propose a semiparametric method for fitting the tail of a heavy-tailed population given a relatively small sample from that population and a larger sample from a related background population. We model the tail of the small sample as an exponential tilt of the better-observed large-sample tail, using a robust sufficient statistic motivated by extreme value theory. In particular, our method induces an estimator of the small-population mean, and we give theoretical and empirical evidence that this estimator outperforms methods that do not use the background sample. We demonstrate substantial …
A Semiparametric Functional Generalized Linear Model for Multi-Population Data
Sauder School of Business, University of British Columbia（不列颠哥伦比亚大学 Sauder 商学院）；Columbia University（哥伦比亚大学）；Department of Statistics and Actuarial Science, Simon Fraser University（西蒙菲莎大学统计与精算科学系）
The density ratio model (DRM) provides a flexible and useful platform for combining information from multiple sources. In this paper, we consider statistical inference under two-sample DRMs with additional parameters defined through and/or additional auxiliary information expressed as estimating equations. We examine the asymptotic properties of the maximum empirical likelihood estimators (MELEs) of the unknown parameters in the DRMs and/or defined through estimating equations, and establish the chi-square limiting distributions for the empirical likelihood ratio (ELR) statistics. We show that…
Parameter shrinkage applied optimally can always reduce error and projection variances from those of maximum likelihood estimation. Many variables that actuaries use are on numerical scales, like age or year, which require parameters at each point. Rather than shrinking these towards zero, nearby parameters are better shrunk towards each other. Semiparametric regression is a statistical discipline for building curves across parameter classes using shrinkage methodology. It is similar to but more parsimonious than cubic splines. We introduce it in the context of Bayesian shrinkage and apply it …
When the response mechanism is believed to be not missing at random (NMAR), a valid analysis requires stronger assumptions on the response mechanism than standard statistical methods would otherwise require. Semiparametric estimators have been developed under the model assumptions on the response mechanism. In this paper, a new statistical test is proposed to guarantee model identifiability without using any instrumental variable. Furthermore, we develop optimal semiparametric estimation for parameters such as the population mean. Specifically, we propose two semiparametric optimal estimators …
We study semiparametric time series models with innovations following a log-concave distribution. We propose a general maximum likelihood framework which allows us to estimate simultaneously the parameters of the model and the density of the innovations. This framework can be easily adapted to many well-known models, including ARMA, GARCH and ARMA-GARCH. Furthermore, we show that the estimator under our new framework is consistent in both ARMA and ARMA-GARCH settings. We demonstrate its finite sample performance via a thorough simulation study and apply it to model the daily log-return of FTSE…
A model-assisted semiparametric method of estimating finite population totals is investigated to improve the precision of survey estimators by incorporating multivariate auxiliary information. The proposed superpopulation model is a single-index model which has proven to be a simple and efficient semiparametric tool in multivariate regression. A class of estimators based on polynomial spline regression is proposed. These estimators are robust against deviation from single-index models. Under standard design conditions, the proposed estimators are asymptotically design-unbiased, consistent and …
Standard logistic regression analysis of case-control data has low power to detect gene-environment interactions, but until recently it was the only method that could be used on complex polygenic data for which parametric distributional models are not feasible. Under the assumption of gene-environment independence in the underlying population, Stalder et al. (2017, Biometrika, 104, 801-812) developed a retrospective method that treats both genetic and environmental variables nonparametrically. However, the mathematical symmetry of genetic and environmental variables is overlooked. We propose a…
Penetrance, which plays a key role in genetic research, is defined as the proportion of individuals with the genetic variants (i.e., {genotype}) that cause a particular trait and who have clinical symptoms of the trait (i.e., {phenotype}). We propose a Bayesian semiparametric approach to estimate the cancer-specific age-at-onset penetrance in the presence of the competing risk of multiple cancers. We employ a Bayesian semiparametric competing risk model to model the duration until individuals in a high-risk group develop different cancers, and accommodate family data using family-wise likeliho…
Missing data occur frequently in empirical studies in health and social sciences, often compromising our ability to make accurate inferences. An outcome is said to be missing not at random (MNAR) if, conditional on the observed variables, the missing data mechanism still depends on the unobserved outcome. In such settings, identification is generally not possible without imposing additional assumptions. Identification is sometimes possible, however, if an instrumental variable (IV) is observed for all subjects which satisfies the exclusion restriction that the IV affects the missingness proces…
Multivariate nonnegative orthant data are real vectors bounded to the left by the null vector, and they can be continuous, discrete or mixed. We first review the recent relative variability indexes for multivariate nonnegative continuous and count distributions. As a prelude, the classification of two comparable distributions having the same mean vector is done through under-, equi- and over-variability with respect to the reference distribution. Multivariate associated kernel estimators are then reviewed with new proposals that can accommodate any nonnegative orthant dataset. We focus on band…
The widespread availability of high-dimensional biological data has made the simultaneous screening of many biological characteristics a central problem in computational biology and allied sciences. While the dimensionality of such datasets continues to grow, so too does the complexity of biomarker identification from exposure patterns in health studies measuring baseline confounders; moreover, doing so while avoiding model misspecification remains an issue only partially addressed. Efficient estimators capable of incorporating flexible, data adaptive regression techniques in estimating releva…
Doubly robust target inference for generalized linear regression with completely missing covariates
School of Mathematics and Statistics, Huazhong University of Science and Technology（华中科技大学数学与统计学院）；Department of Statistics and Data Science, Tsinghua University（清华大学统计与数据科学系）
We propose a class of locally and asymptotically optimal tests, based on multivariate ranks and signs for the homogeneity of scatter matrices in $m$ elliptical populations. Contrary to the existing parametric procedures, these tests remain valid without any moment assumptions, and thus are perfectly robust against heavy-tailed distributions (validity robustness). Nevertheless, they reach semiparametric efficiency bounds at correctly specified elliptical densities and maintain high powers under all (efficiency robustness). In particular, their normal-score version outperforms traditional Gaussi…
Doubly Robust Machine Learning for Population Size Estimation with Missing Covariates: Application to Gaza Conflict Mortality
Center for Data Science New York University（纽约大学数据科学中心）；Department of Statistics & Data Science Carnegie Mellon University（卡内基梅隆大学统计与数据科学系）；Department of Medical Statistics London School of Hygiene and Tropical Medicine（伦敦卫生与热带医学院医学统计学系）
This paper proposes using a method named Double Score Matching (DSM) to do mass-imputation and presents an application to make inferences with a nonprobability sample. DSM is a $k$-Nearest Neighbors algorithm that uses two balance scores instead of covariates to reduce the dimension of the distance metric and thus to achieve a faster convergence rate. DSM mass-imputation and population inference are consistent if one of two balance score models is correctly specified. Simulation results show that the DSM performs better than recently developed double robust estimators when the data generating …
Debiased Machine Learning for Conformal Prediction of Counterfactual Outcomes Under Runtime Confounding
Department of Biostatistics, Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院生物统计系）；Department of Biostatistics and Informatics, Colorado School of Public Health（科罗拉多公共卫生学院生物统计与信息学系）；Department of Data Science, Dana Farber Cancer Institute（丹娜-法伯癌症研究所数据科学系）
We consider incomplete observations of stochastic processes governing the spread of infectious diseases through finite populations by way of contact. We propose a flexible semiparametric modeling framework with at least three advantages. First, it enables researchers to study the structure of a population contact network and its impact on the spread of infectious diseases. Second, it can accommodate short- and long-tailed degree distributions and detect potential superspreaders, who represent an important public health concern. Third, it addresses the important issue of incomplete data. Starti…
Area Under the Curve (AUC) is arguably the most popular measure of classification accuracy. We use a semiparametric framework to introduce a latent scale-invariant $R^2$, a novel measure of variation explained for an observed binary outcome and an observed continuous predictor, and then directly link the latent $R^2$ to AUC. This enables a mutually consistent simultaneous use of AUC as a measure of classification accuracy and the latent $R^2$ as a scale-invariant measure of explained variation. Specifically, we employ Semiparametric Gaussian Copula (SGC) to model a joint dependence between obs…
In this paper, we propose a general subgroup analysis framework based on semiparametric additive mixed effect models in longitudinal analysis, which can identify subgroups on each covariate and estimate the corresponding regression functions simultaneously. In addition, the proposed procedure is applicable for both balanced and unbalanced longitudinal data. A backfitting combined with k-means algorithm is developed to estimate each semiparametric additive component across subgroups and detect subgroup structure on each covariate respectively. The actual number of groups is estimated by minimiz…
In this paper we introduce a mixture cure model with a linear hazard rate regression model for the event times. Cure models are statistical models for event times that take into account that a fraction of the population might never experience the event of interest, this fraction is said to be {`}cured{'}. The population survival function in a mixture cure model takes the form $S(t) = 1 - π+ π\exp(-\int_0^tα(s)\,d s)$, where $π$ is the probability of being susceptible to the event under study, and $α(s)$ is the hazard rate of the susceptible fraction. We let both $π$ and $α(s)$ depend on possib…
Time-series data in population health and epidemiology often involve non-Gaussian responses. In this note, we propose a semiparametric generalized linear models framework for time-series data that does not require specification of a working conditional response distribution for the data. Instead, the underlying response distribution is treated as an infinite-dimensional parameter which is estimated simultaneously with the usual finite-dimensional parameters via a maximum empirical likelihood approach. A general consistency result for the resulting estimators is given. Simulations suggest that …
Suppose one is interested in estimating causal effects in the presence of potentially unmeasured confounding with the aid of a valid instrumental variable. This paper investigates the problem of making inferences about the average treatment effect when data are fused from two separate sources, one of which contains information on the treatment and the other contains information on the outcome, while values for the instrument and a vector of baseline covariates are recorded in both. We provide a general set of sufficient conditions under which the average treatment effect is nonparametrically i…
Standard methods for inference about direct and indirect effects require stringent no unmeasured confounding assumptions which often fail to hold in practice, particularly in observational studies. The goal of this paper is to introduce a new form of indirect effect, the population intervention indirect effect (PIIE), that can be nonparametrically identified in the presence of an unmeasured common cause of exposure and outcome. This new type of indirect effect captures the extent to which the effect of exposure is mediated by an intermediate variable under an intervention that holds the compon…
Trials enroll a large number of subjects in order to attain power, making them expensive and time-consuming. Sample size calculations are often performed with the assumption of an unadjusted analysis, even if the trial analysis plan specifies a more efficient estimator (e.g. ANCOVA). This leads to conservative estimates of required sample sizes and an opportunity for savings. Here we show that a relatively simple formula can be used to estimate the power of any two-arm, single-timepoint trial analyzed with a semiparametric efficient estimator, regardless of the domain of the outcome or kind of…
A new and an enriched JPEG algorithm is provided for identifying redundancies in a sequence of irregular noisy data points which also accommodates a reference-free criterion function. Our main contribution is by formulating analytically (instead of approximating) the inverse of the transpose of JPEGwavelet transform without involving matrices which are computationally cumbersome. The algorithm is suitable for the widely-spread situations where the original data distribution is unobservable such as in cases where there is deficient representation of the entire population in the training data (i…
Augmented transfer regression learning for completely missing covariates
School of Mathematics and Statistics, Huazhong University of Science and Technology（华中科技大学数学与统计学院）；Department of Statistics, Colorado State University（科罗拉多州立大学统计学系）
Causal inference in a program evaluation setting faces the problem of external validity when the treatment effect in the target population is different from the treatment effect identified from the population of which the sample is representative. This paper focuses on a situation where such discrepancy arises by a stratified sampling design based on the individual treatment status and other characteristics. In such settings, the design probability is known from the sampling design but the target population depends on the underlying population share vector which is often unknown, and except fo…
In the presence of heterogeneity between the randomized controlled trial (RCT) participants and the target population, evaluating the treatment effect solely based on the RCT often leads to biased quantification of the real-world treatment effect. To address the problem of lack of generalizability for the treatment effect estimated by the RCT sample, we leverage observational studies with large samples that are representative of the target population. This paper concerns evaluating treatment effects on survival outcomes for a target population and considers a broad class of estimands that are …
Generalizing conditional average treatment effects from nested randomized trials to all trial-eligible individuals
University of Waterloo（滑铁卢大学）；Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）；Brown University School of Public Health（布朗大学公共卫生学院）；Center for Gerontology & Healthcare Research, Brown University School of Public Health（布朗大学公共卫生学院老年学与医疗保健研究中心）
