Extending Mixture of Experts Model to Investigate Heterogeneity of Trajectories: When, Where and How to Add Which Covariates
Jin Liu, Robert A. Perera
备注：
\textcircled{c}2021, American Psychological Association. This paper is not the copy of record and may not exactly replicate the final, authoritative version of the article. Please do not copy or cite without authors' permission. The final article will be available, upon publication, via its DOI: 10.1037/met0000436
Researchers are usually interested in examining the impact of covariates when separating heterogeneous samples into latent classes that are more homogeneous. The majority of theoretical and empirical studies with such aims have focused on identifying covariates as predictors of class membership in the structural equation modeling framework. In other words, the covariates only indirectly affect the sample heterogeneity. However, the covariates' influence on between-individual differences can also be direct. This article presents a mixture model that investigates covariates to explain within-cluster and between-cluster heterogeneity simultaneously, known as a mixture-of-experts (MoE) model. This study aims to extend the MoE framework to investigate heterogeneity in nonlinear trajectories: to identify latent classes, covariates as predictors to clusters, and covariates that explain within-cluster differences in change patterns over time. Our simulation studies demonstrate that the proposed model generally estimates the parameters unbiasedly, precisely and exhibits appropriate empirical coverage for a nominal 95% confidence interval. This study also proposes implementing structural equation model forests to shrink the covariate space of the proposed mixture model. We illustrate how to select covariates and construct the proposed model with longitudinal mathematics achievement data. Additionally, we demonstrate that the proposed mixture model can be further extended in the structural equation modeling framework by allowing the covariates that have direct effects to be time-varying.
The paper describes a new class of capture-recapture models for closed populations when individual covariates are available. The novelty consists in combining a latent class model for the distribution of the capture history, where the class weights and the conditional distributions given the latent may depend on covariates, with a model for the marginal distribution of the available covariates as in \cite{Liu2017}. In addition, any general form of serial dependence is allowed when modeling capture histories conditionally on the latent and covariates. A Fisher-scoring algorithm for maximum like…
Networks are often characterized by node heterogeneity for which nodes exhibit different degrees of interaction and link homophily for which nodes sharing common features tend to associate with each other. In this paper, we propose a new directed network model to capture the former via node-specific parametrization and the latter by incorporating covariates. In particular, this model quantifies the extent of heterogeneity in terms of outgoingness and incomingness of each node by different parameters, thus allowing the number of heterogeneity parameters to be twice the number of nodes. We study…
An important phenomenon in high dimensional biological data is the presence of unobserved covariates that can have a significant impact on the measured response. When these factors are also correlated with the covariate(s) of interest (i.e. disease status), ignoring them can lead to increased type I error and spurious false discovery rate estimates. We show that depending on the strength of this correlation and the informativeness of the observed data for the latent factors, previously proposed estimators for the effect of the covariate of interest that attempt to account for unobserved covari…
Due to patient heterogeneity in response to various aspects of any treatment program, biomedical and clinical research is gradually shifting from the traditional "one-size-fits-all" approach to the new paradigm of personalized medicine. An important step in this direction is to identify the treatment by covariate interactions. We consider the setting in which there are potentially a large number of covariates of interest. Although a number of novel machine learning methodologies have been developed in recent years to aid in treatment selection in this setting, few, if any, have adopted formal …
Network (or matrix) reconstruction is a general problem which occurs if the margins of a matrix are given and the matrix entries need to be predicted. In this paper we show that the predictions obtained from the iterative proportional fitting procedure (IPFP) or equivalently maximum entropy (ME) can be obtained by restricted maximum likelihood estimation relying on augmented Lagrangian optimization. Based on the equivalence we extend the framework of network reconstruction towards regression by allowing for exogenous covariates and random heterogeneity effects. The proposed estimation approach…
Based on structured data derived from large complex systems, we computationally further develop and refine a major factor selection protocol by accommodating structural dependency and heterogeneity among many features to unravel data's information content. Two operational concepts: ``de-associating'' and its counterpart ``shadowing'' that play key roles in our protocol, are reasoned, explained, and carried out via contingency table platforms. This protocol via ``de-associating'' capability would manifest data's information content by identifying which covariate feature-sets do or don't provide…
In many complex applications, data heterogeneity and homogeneity exist simultaneously. Ignoring either one will result in incorrect statistical inference. In addition, coping with complex data that are non-Euclidean becomes more common. To address these issues we consider a distributional data response additive model in which the response is a distributional density function and the individual effect curves are homogeneous within a group but heterogeneous across groups, the covariates capturing the variation share common additive bivariate functions. A transformation approach is first utilized…
Two-stage least squares with treatment-covariate interactions for treatment effect heterogeneity
Fuqua School of Business, Duke University（杜克大学福库商学院）；Department of Statistics, University of California, Berkeley（加州大学伯克利分校统计学系）；Department of Statistical Science, Duke University（杜克大学统计科学系）
The edges in networks are not only binary, either present or absent, but also take weighted values in many scenarios (e.g., the number of emails between two users). The covariate-$p_0$ model has been proposed to model binary directed networks with the degree heterogeneity and covariates. However, it may cause information loss when it is applied in weighted networks. In this paper, we propose to use the Poisson distribution to model weighted directed networks, which admits the sparsity of networks, the degree heterogeneity and the homophily caused by covariates of nodes. We call it the \emph{ne…
We propose a general model that jointly characterizes degree heterogeneity and homophily in weighted, undirected networks. We present a moment estimation method using node degrees and homophily statistics. We establish consistency and asymptotic normality of our estimator using novel analysis. We apply our general framework to three applications, including both exponential family and non-exponential family models. Comprehensive numerical studies and a data example also demonstrate the usefulness of our method.
In this paper we consider the estimation of population size from one-source capture--recapture data, that is, a list in which individuals can potentially be found repeatedly and where the question is how many individuals are missed by the list. As a typical example, we provide data from a drug user study in Bangkok from 2001 where the list consists of drug users who repeatedly contact treatment institutions. Drug users with 1, 2, 3$,...$ contacts occur, but drug users with zero contacts are not present, requiring the size of this group to be estimated. Statistically, these data can be consider…
An important task in drug development is to identify patients, which respond better or worse to an experimental treatment. Identifying predictive covariates, which influence the treatment effect and can be used to define subgroups of patients, is a key aspect of this task. Analyses of treatment effect heterogeneity are however known to be challenging, since the number of possible covariates or subgroups is often large, while samples sizes in earlier phases of drug development are often small. In addition, distinguishing predictive covariates from prognostic covariates, which influence the resp…
Researchers continue to be interested in exploring the effects that covariates have on the heterogeneity in trajectories. The inclusion of covariates associated with latent classes allows for a more clear understanding of individual differences and a more meaningful interpretation of latent class membership. Many theoretical and empirical studies have focused on investigating heterogeneity in change patterns of a univariate repeated outcome and examining the effects on baseline covariates that inform the cluster formation. However, developmental processes rarely unfold in isolation; therefore,…
Empirical researchers are usually interested in investigating the impacts of baseline covariates have when uncovering sample heterogeneity and separating samples into more homogeneous groups. However, a considerable number of studies in the structural equation modeling (SEM) framework usually start with vague hypotheses in terms of heterogeneity and possible reasons. It suggests that (1) the determination and specification of a proper model with covariates is not straightforward, and (2) the exploration process may be computational intensive given that a model in the SEM framework is usually c…
Including covariates in loglinear models of population registers improves population size estimates for two reasons. First, it is possible to take heterogeneity of inclusion probabilities over the levels of a covariate into account; and second, it allows subdivision of the estimated population by the levels of the covariates, giving insight into characteristics of individuals that are not included in any of the registers. The issue of whether or not marginalizing the full table of registers by covariates over one or more covariates leaves the estimated population size estimate invariant is int…
Covariate Informed Identification of Heterogeneity and Outliers in Longitudinal Data
University of Louisville（路易斯维尔大学）；Case Western Reserve University（凯斯西储大学）；Mass General Brigham, Harvard Medical School（麻省总医院布里格姆医院，哈佛医学院）；University of Missouri（密苏里大学）
Products manufactured from the same batch or utilized in the same region often exhibit correlated lifetime observations due to the latent heterogeneity caused by the influence of shared but unobserved covariates. The unavailable group-shared covariates involve multiple different types (e.g., discrete, continuous, or mixed-type) and induce different structures of indispensable group-shared latent heterogeneity. Without carefully capturing such latent heterogeneity, the lifetime modeling accuracy will be significantly undermined. In this work, we propose a generic Bayesian lifetime modeling appr…
This paper proposes a novel method to estimate large panel data error-correction models with stationary/non-stationary covariates and spatially dependent errors, which allows for known/unknown group-specific patterns of slope heterogeneity. Analysis is based on composite quasi-likelihood (CQL) maximization which performs estimation and classification simultaneously. The proposed CQL estimator remains unbiased in the presence of misspecification of the unobserved individual/group-specific fixed effects; therefore, neither instrumental variables nor bias corrections/reductions are required. This…
Dissecting Microbial Community Structure and Heterogeneity via Multivariate Covariate-Adjusted Clustering
University of Connecticut Health Center（康涅狄格大学健康中心）；Department of Biostatistics, University of Michigan（密歇根大学生物统计系）；Department of Statistics, University of Connecticut（康涅狄格大学统计学系）
Multi-Task Learning with Covariate-Overlap Regularization
School of Statistics and Data Science, Shanghai University of Finance and Economics（上海财经大学统计与数据科学学院）；Department of Statistics and Data Science, Carnegie Mellon University（卡内基梅隆大学统计与数据科学系）；Department of Statistics, University of California, Irvine（加州大学 Irvine 分校统计系）
Spatial regression is widely used for modeling the relationship between a dependent variable and explanatory covariates. Oftentimes, the linear relationships vary across space, when some covariates have location-specific effects on the response. One fundamental question is how to detect the systematic variation in the model and identify which locations share common regression coefficients and which do not. Only a correct model structure can assure unbiased estimation of coefficients and valid inferences. In this work, we propose a new procedure, called Spatial Heterogeneity Automatic Detection…
In this paper, we propose a doubly robust method to present the heterogeneity of the average treatment effect with respect to observed covariates of interest. We consider a situation where a large number of covariates are needed for identifying the average treatment effect but the covariates of interest for analyzing heterogeneity are of much lower dimension. Our proposed estimator is doubly robust and avoids the curse of dimensionality. We propose a uniform confidence band that is easy to compute, and we illustrate its usefulness via Monte Carlo experiments and an application to the effects o…
Cluster-induced target shift and synthetic approximation for high-dimensional clustered data
Florida International University（佛罗里达国际大学）；University of Nottingham Ningbo China（诺丁汉大学宁波校区）；University of Pennsylvania（宾夕法尼亚大学）；Dartmouth College（达特茅斯学院）；Oklahoma State University（俄克拉荷马州立大学）
In many biomedical problems, data are often heterogeneous, with samples spanning multiple patient subgroups, where different subgroups may have different disease subtypes, stages, or other medical contexts. These subgroups may be related, but they are also expected to have differences with respect to the underlying biology. The heterogeneous data presents a precious opportunity to explore the heterogeneities and commonalities between related subgroups. Unfortunately, effective statistical analysis methods are still lacking. Recently, several novel methods based on integrative analysis have bee…
We consider the problem of selecting the optimal subgroup to treat when data on covariates is available from a randomized trial or observational study. We distinguish between four different settings including (i) treatment selection when resources are constrained, (ii) treatment selection when resources are not constrained, (iii) treatment selection in the presence of side effects and costs, and (iv) treatment selection to maximize effect heterogeneity. We show that, in each of these cases, the optimal treatment selection rule involves treating those for whom the predicted mean difference in o…
We provide a simple formulation of the conditions under which ecological bias should be expected and argue that the bias will affect any method of ecological inference; our claim is supported by formal derivations and several examples where individual data are available. The conditions which we highlight imply that, when they are violated, ecological bias cannot be avoided unless the a suitable model for the effect of specific covariates is incorporated into ecological inference. We also detect situations where the ecological bias cannot be corrected even if the effect of covariates is incorpo…
Generalizing estimates of causal effects from an experiment to a target population is of interest to scientists. However, researchers are usually constrained by available covariate information. Analysts can often collect much fewer variables from population samples than from experimental samples, which has limited applicability of existing approaches that assume rich covariate data from both experimental and population samples. In this article, we examine how to select covariates necessary for generalizing experimental results under such data constraints. In our concrete context of a large-sca…
In a regression analysis, suppose we suspect that there are several heterogeneous groups in the population that a sample represents. Mixture regression models have been applied to address such problems. By modeling the conditional distribution of the response given the covariate as a mixture, the sample can be clustered into groups and the individual regression models for the groups can be estimated simultaneously. This approach treats the covariate as deterministic so that the covariate carries no information as to which group the subject is likely to belong to. Although this assumption may b…
Mixtures of experts models provide a framework in which covariates may be included in mixture models. This is achieved by modelling the parameters of the mixture model as functions of the concomitant covariates. Given their mixture model foundation, mixtures of experts models possess a diverse range of analytic uses, from clustering observations to capturing parameter heterogeneity in cross-sectional data. This chapter focuses on delineating the mixture of experts modelling framework and demonstrates the utility and flexibility of mixtures of experts models as an analytic tool.
Principal Stratification with Bayesian Additive Regression Trees for Count-Valued Intermediate Variables: Estimating the Effect of Fertility on Women's Employment
Domain Adaptation Targeting Heterogeneous and Imbalanced Subgroups
National University of Singapore（新加坡国立大学）；Bentley University（本特利大学）；Peking University Health Science Center（北京大学健康科学中心）；Harvard T.H. Chan School of Public Health and Harvard Medical School（哈佛大学陈曾熙公共卫生学院和哈佛医学院）；Peking University（北京大学）
Meta-analysis, because of both logistical convenience and statistical efficiency, is widely popular for synthesizing information on common parameters of interest across multiple studies. We propose developing a generalized meta-analysis approach for combining information on multivariate regression parameters across multiple different studies which have varying level of covariate information. Using algebraic relationships between regression parameters in different dimensions, we specify a set of moment equations for estimating parameters of a maximal model through information available from set…
Spatial point pattern data are routinely encountered. A flexible regression model for the underlying intensity is essential to characterizing the spatial point pattern and understanding the impacts of potential risk factors on such pattern. We propose a Bayesian semiparametric regression model where the observed spatial points follow a spatial Poisson process with an intensity function which adjusts a nonparametric baseline intensity with multiplicative covariate effects. The baseline intensity is piecewise constant, approached with a powered Chinese restaurant process prior which prevents an …
We consider the problem of causal discovery (structure learning) from heterogeneous observational data. Most existing methods assume a homogeneous sampling scheme, which leads to misleading conclusions when violated in many applications. To this end, we propose a novel approach that exploits data heterogeneity to infer possibly cyclic causal structures from causally insufficient systems. The core idea is to model the direct causal effects as functions of exogenous covariates that properly explain data heterogeneity. We investigate structure identifiability properties of the proposed model. Str…
