Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: Assumptions, models, effect scales, data scenarios, and implementation details
Trang Quynh Nguyen, Benjamin Ackerman, Ian Schmid, Stephen R. Cole, Elizabeth A. Stuart
Background: Randomized controlled trials are often used to inform policy and practice for broad populations. The average treatment effect (ATE) for a target population, however, may be different from the ATE observed in a trial if there are effect modifiers whose distribution in the target population is different that from that in the trial. Methods exist to use trial data to estimate the target population ATE, provided the distributions of treatment effect modifiers are observed in both the trial and target population -- an assumption that may not hold in practice.
Methods: The proposed sensitivity analyses address the situation where a treatment effect modifier is observed in the trial but not the target population. These methods are based on an outcome model or the combination of such a model and weighting adjustment for observed differences between the trial sample and target population. They accommodate several types of outcome models: linear models (including single time outcome and pre- and post-treatment outcomes) for additive effects, and models with log or logit link for multiplicative effects. We clarify the methods' assumptions and provide detailed implementation instructions.
Illustration: We illustrate the methods using an example generalizing the effects of an HIV treatment regimen from a randomized trial to a relevant target population.
Conclusion: These methods allow researchers and decision-makers to have more appropriate confidence when drawing conclusions about target population effects.
Sparse additive modeling is a class of effective methods for performing high-dimensional nonparametric regression. This paper develops a sparse additive model focused on estimation of treatment effect-modification with simultaneous treatment effect-modifier selection. We propose a version of the sparse additive model uniquely constrained to estimate the interaction effects between treatment and pretreatment covariates, while leaving the main effects of the pretreatment covariates unspecified. The proposed regression model can effectively identify treatment effect-modifiers that exhibit possibl…
When individuals participating in a randomized trial differ with respect to the distribution of effect modifiers compared compared with the target population where the trial results will be used, treatment effect estimates from the trial may not directly apply to target population. Methods for extending -- generalizing or transporting -- causal inferences from the trial to the target population rely on conditional exchangeability assumptions between randomized and non-randomized individuals. The validity of these assumptions is often uncertain or controversial and investigators need to examine…
We consider methods for causal inference in randomized trials nested within cohorts of trial-eligible individuals, including those who are not randomized. We show how baseline covariate data from the entire cohort, and treatment and outcome data only from randomized individuals, can be used to identify potential (counterfactual) outcome means and average treatment effects in the target population of all eligible individuals. We review identifiability conditions, propose estimators, and assess the estimators' finite-sample performance in simulation studies. As an illustration, we apply the esti…
Randomized controlled trials (RCTs) provide strong internal validity compared with observational studies. However, selection bias threatens the external validity of randomized trials. Thus, RCT results may not apply to either broad public policy populations or narrow populations, such as specific insurance pools. Some researchers use propensity scores (PSs) to generalize results from an RCT to a target population. In this scenario, a PS is defined as the probability of participating in the trial conditioning on observed covariates. We study a model-free inverse probability weighted estimator (…
Estimating Causal Treatment Effects in Placebo-Controlled Randomized Clinical Trials When High Placebo Response is Anticipated
Smith Center for Outcomes Research in Cardiology, Beth Israel Deaconess Medical Center（贝斯以色列女医生医疗中心心脏结局研究史密斯中心）；Department of Statistics, SungKyunKwan University（成均馆大学统计系）；Biostatistics Department, Boston University（波士顿大学生物统计学系）
When treatment effect modifiers influence the decision to participate in a randomized trial, the average treatment effect in the population represented by the randomized individuals will differ from the effect in other populations. In this tutorial, we consider methods for extending causal inferences about time-fixed treatments from a trial to a new target population of non-participants, using data from a completed randomized trial and baseline covariate data from a sample from the target population. We examine methods based on modeling the expectation of the outcome, the probability of partic…
In multicenter randomized trials, when effect modifiers have a different distribution across centers, comparisons between treatment groups that average over centers may not apply to any of the populations underlying the individual centers. Here, we describe methods for reinterpreting the evidence produced by a multicenter trial in the context of the population underlying each center. We describe how to identify center-specific effects under identifiability conditions that are largely supported by the study design and when associations between center membership and the outcome may be present, g…
When the distribution of treatment effect modifiers differs between the trial sample and target population, inverse probability weighting (IPSW) can be applied to achieve an unbiased estimate of the population average treatment effect in the target population. The statistical validity of IPSW is threatened when there are missing data in the target population, including potential missingness in trial sample. However, missing data methods have not been adequately discussed in the current literature. We conducted a set of simulation studies to determine how to apply multiple imputation (MI) in th…
The ability to generalize experimental results from randomized control trials (RCTs) across locations is crucial for informing policy decisions in targeted regions. Such generalization is often hindered by the lack of identifiability due to unmeasured effect modifiers that compromise direct transport of treatment effect estimates from one location to another. We build upon sensitivity analysis in observational studies and propose an optimization procedure that allows us to get bounds on the treatment effects in targeted regions. Furthermore, we construct more informative bounds by balancing on…
Randomized trials are considered the gold standard for estimating causal effects. Trial findings are often used to inform policy and programming efforts, yet their results may not generalize well to a relevant target population due to potential differences in effect moderators between the trial and population. Statistical methods have been developed to improve generalizability by combining trials and population data, and weighting the trial to resemble the population on baseline covariates.Large-scale surveys in fields such as health and education with complex survey designs are a logical sour…
When engagement with a randomized trial is driven by factors that affect the outcome or when trial engagement directly affects the outcome independent of treatment, the average treatment effect among trial participants is unlikely to generalize to a target population. In this paper, we use counterfactual and graphical causal models to examine under what conditions we can generalize causal inferences from a randomized trial to the target population of trial-eligible individuals. We offer an interpretation of generalizability analyses using the notion of a hypothetical intervention to "scale-up"…
Individual participant data (IPD) meta-analyses of randomised trials are considered a reliable way to assess participant-level treatment effect modifiers but may not make the best use of the available data. Traditionally, effect modifiers are explored one covariate at a time, which gives rise to the possibility that evidence of treatment-covariate interaction may be due to confounding from a different, related covariate. We aimed to evaluate current practice when estimating treatment-covariate interactions in IPD meta-analysis, specifically focusing on involvement of additional covariates in t…
We show how entropy balancing can be used for transporting experimental treatment effects from a trial population onto a target population. This method is doubly-robust in the sense that if either the outcome model or the probability of trial participation is correctly specified, then the estimate of the target population average treatment effect is consistent. Furthermore, we only require the sample moments of the effect modifiers drawn from the target population to consistently estimate the target population average treatment effect. We compared the finite-sample performance of entropy balan…
In the presence of heterogeneity between the randomized controlled trial (RCT) participants and the target population, evaluating the treatment effect solely based on the RCT often leads to biased quantification of the real-world treatment effect. To address the problem of lack of generalizability for the treatment effect estimated by the RCT sample, we leverage observational studies with large samples that are representative of the target population. This paper concerns evaluating treatment effects on survival outcomes for a target population and considers a broad class of estimands that are …
Extending (generalizing or transporting) causal inferences from a randomized trial to a target population requires ``generalizability'' or ``transportability'' assumptions, which state that randomized and non-randomized individuals are exchangeable conditional on baseline covariates. These assumptions are made on the basis of background knowledge, which is often uncertain or controversial, and need to be subjected to sensitivity analysis. We present simple methods for sensitivity analyses that do not require detailed background knowledge about specific unknown or unmeasured determinants of the…
Recent work has made important contributions in the development of causally-interpretable meta-analysis. These methods transport treatment effects estimated in a collection of randomized trials to a target population of interest. Ideally, estimates targeted toward a specific population are more interpretable and relevant to policy-makers and clinicians. However, between-study heterogeneity not arising from differences in the distribution of treatment effect modifiers can raise difficulties in synthesizing estimates across trials. The existence of such heterogeneity, including variations in tre…
In vaccine studies, investigators are often interested in studying effect modifiers of clinical treatment efficacy by biomarker-based principal strata, which is useful for selecting biomarker study endpoints for evaluating treatments in new trials, exploring biological mechanisms of clinical treatment efficacy, and studying mediators of clinical treatment efficacy. However, in trials where participants may enter the study with prior exposure therefore with variable baseline biomarker values, clinical treatment efficacy may depend jointly on a biomarker measured at baseline and measured at a fi…
Methods for extending -- generalizing or transporting -- inferences from a randomized trial to a target population involve conditioning on a large set of covariates that is sufficient for rendering the randomized and non-randomized groups exchangeable. Yet, decision-makers are often interested in examining treatment effects in subgroups of the target population defined in terms of only a few discrete covariates. Here, we propose methods for estimating subgroup-specific potential outcome means and average treatment effects in generalizability and transportability analyses, using outcome model-b…
Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large coho…
We examine study designs for extending (generalizing or transporting) causal inferences from a randomized trial to a target population. Specifically, we consider nested trial designs, where randomized individuals are nested within a sample from the target population, and non-nested trial designs, including composite dataset designs, where a randomized trial is combined with a separately obtained sample of non-randomized individuals from the target population. We show that the causal quantities that can be identified in each study design depend on what is known about the probability of sampling…
Estimation of conditional average treatment effects (CATEs) plays an essential role in modern medicine by informing treatment decision-making at a patient level. Several metalearners have been proposed recently to estimate CATEs in an effective and flexible way by re-purposing predictive machine learning models for causal estimation. In this chapter, we summarize the literature on metalearners and provide concrete guidance for their application for treatment heterogeneity estimation from randomized controlled trials' data with survival outcomes. The guidance we provide is supported by a compre…
Prediction-powered Generalization of Causal Inferences
Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard（埃里克和温迪·施密特中心，麻省理工学院与哈佛大学博德研究所）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Department of Computational Precision Health, UC Berkeley and UCSF（加州大学伯克利分校与加州大学旧金山分校计算精准健康系）
Investigators are increasingly using novel methods for extending (generalizing or transporting) causal inferences from a trial to a target population. In many generalizability and transportability analyses, the trial and the observational data from the target population are separately sampled, following a non-nested trial design. In practical implementations of this design, non-randomized individuals from the target population are often identified by conditioning on the use of a particular treatment, while individuals who used other candidate treatments for the same indication or individuals w…
Reframing Population-Adjusted Indirect Comparisons as a Transportability Problem: An Estimand-Based Perspective and Implications for Health Technology Assessment
Methods for extending -- generalizing or transporting -- inferences from a randomized trial to a target population involve conditioning on a large set of covariates that is sufficient for rendering the randomized and non-randomized groups exchangeable. Yet, decision-makers are often interested in examining treatment effects in subgroups of the target population defined in terms of only a few discrete covariates. Here, we propose methods for estimating subgroup-specific potential outcome means and average treatment effects in generalizability and transportability analyses, using outcome model-b…
We propose a test-based elastic integrative analysis of the randomized trial and real-world data to estimate treatment effect heterogeneity with a vector of known effect modifiers. When the real-world data are not subject to bias, our approach combines the trial and real-world data for efficient estimation. Utilizing the trial design, we construct a test to decide whether or not to use real-world data. We characterize the asymptotic distribution of the test-based estimator under local alternatives. We provide a data-adaptive procedure to select the test threshold that promises the smallest mea…
Randomized Controlled Trials (RCTs) are often considered the gold standard for estimating causal effect, but they may lack external validity when the population eligible to the RCT is substantially different from the target population. Having at hand a sample of the target population of interest allows us to generalize the causal effect. Identifying the treatment effect in the target population requires covariates to capture all treatment effect modifiers that are shifted between the two sets. Standard estimators then use either weighting (IPSW), outcome modeling (G-formula), or combine the tw…
It is known that the hazard ratio lacks a useful causal interpretation. Even for data from a randomized controlled trial, the hazard ratio suffers from built-in selection bias as, over time, the individuals at risk in the exposed and unexposed are no longer exchangeable. In this work, we formalize how the observed hazard ratio evolves and deviates from the causal hazard ratio of interest in the presence of heterogeneity of the hazard of unexposed individuals (frailty) and heterogeneity in effect (individual modification). For the case of effect heterogeneity, we define the causal hazard ratio.…
We aim to generalize the results of a randomized controlled trial (RCT) to a target population with the help of some observational data. This is a problem of causal effect identification with multiple data sources. Challenges arise when the RCT is conducted in a context that differs from the target population. Earlier research has focused on cases where the estimates from the RCT can be adjusted by observational data in order to remove the selection bias and other domain specific differences. We consider examples where the experimental findings cannot be generalized by an adjustment and show t…
We focus on the problem of generalizing a causal effect estimated on a randomized controlled trial (RCT) to a target population described by a set of covariates from observational data. Available methods such as inverse propensity sampling weighting are not designed to handle missing values, which are however common in both data sources. In addition to coupling the assumptions for causal effect identifiability and for the missing values mechanism and to defining appropriate estimation strategies, one difficulty is to consider the specific structure of the data with two sources and treatment an…
To generalize inferences from a randomized trial to the target population of all trial-eligible individuals, investigators can use nested trial designs, where the randomized individuals are nested within a cohort of trial-eligible individuals, including those who are not offered or refuse randomization. In these designs, data on baseline covariates are collected from the entire cohort, and treatment and outcome data need only be collected from randomized individuals. In this paper, we describe nested trial designs that improve research economy by collecting additional baseline covariate data a…
Hazard ratios are prone to selection bias, compromising their use as causal estimands. On the other hand, the hazard difference has been shown to remain unaffected by the selection of frailty factors over time. Therefore, observed hazard differences can be used as an unbiased estimator for the causal hazard differences in the absence of confounding. However, in the presence of effect (on the hazard) heterogeneity, the hazard difference is also affected by selection. In this work, we formalize how the observed hazard difference (from a randomized controlled trial) evolves by selecting favourabl…
We discuss the identifiability of causal estimands for generalizability and transportability analyses, both under perfect and imperfect adherence to treatment assignment. We consider a setting where the trial data contain information on baseline covariates, assignment at baseline, intervention at baseline (point treatment), and outcomes; and where the data from non-randomized individuals only contain information on baseline covariates. In this setting, we review identification results under perfect adherence and study two examples in which non-adherence severely limits the ability to transport…
