Mixtures of regression are a powerful class of models for regression learning with respect to a highly uncertain and heterogeneous response variable of interest. In addition to being a rich predictive model for the response given some covariates, the parameters in this model class provide useful information about the heterogeneity in the data population, which is represented by the conditional distributions for the response given the covariates associated with a number of distinct but latent subpopulations. In this paper, we investigate conditions of strong identifiability, rates of convergence for conditional density and parameter estimation, and the Bayesian posterior contraction behavior arising in finite mixture of regression models, under exact-fitted and over-fitted settings and when the number of components is unknown. This theory is applicable to common choices of link functions and families of conditional distributions employed by practitioners. We provide simulation studies and data illustrations, which shed some light on the parameter learning behavior found in several popular regression mixture models reported in the literature.
In this paper, we introduce a new identifiability criteria for linear structural equation models, which we call regression identifiability. We provide necessary and sufficient graphical conditions for a directed edge to be regression identifiable. Suppose $Σ^*$ corresponds to the covariance matrix of the graphical model $G^*$ obtained by performing an edge intervention to $G$ with corresponding covariance matrix $Σ$. We first obtain necessary and sufficient conditions for $Σ^*$ to be identifiable given $Σ$. Using regression identifiability, we obtain necessary graphical conditions for $Σ^*$ to…
Synthetic Potential Outcomes and Causal Mixture Identifiability
Eric and Wendy Schmidt Center（埃里克和温迪·施密特中心）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学布罗德研究所）；Laboratory for Information and Decision Systems（信息与决策系统实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Gaussian process regression (GPR) model is a popular nonparametric regression model. In GPR, features of the regression function such as varying degrees of smoothness and periodicities are modeled through combining various covarinace kernels, which are supposed to model certain effects. The covariance kernels have unknown parameters which are estimated by the EM-algorithm or Markov Chain Monte Carlo. The estimated parameters are keys to the inference of the features of the regression functions, but identifiability of these parameters has not been investigated. In this paper, we prove identifia…
In the last decade, the secondary use of large data from health systems, such as electronic health records, has demonstrated great promise in advancing biomedical discoveries and improving clinical decision making. However, there is an increasing concern about biases in association studies caused by misclassification in the binary outcomes derived from electronic health records. We revisit the classical logistic regression model with misclassified outcomes. Despite that local identification conditions in some related settings have been previously established, the global identification of such …
Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
Gaussian processes (GPs) are a well-known nonparametric Bayesian inference technique, but they suffer from scalability problems for large sample sizes, and their performance can degrade for non-stationary or spatially heterogeneous data. In this work, we seek to overcome these issues through (i) employing variational free energy approximations of GPs operating in tandem with online expectation propagation steps; and (ii) introducing a local splitting step which instantiates a new GP whenever the posterior distribution changes significantly as quantified by the Wasserstein metric over posterior…
Prediction Markets as Bayesian Inverse Problems: Uncertainty Quantification, Identifiability, and Information Gain from Price-Volume Histories under Latent Types
In economic development, there are often regions that share similar economic characteristics, and economic models on such regions tend to have similar covariate effects. In this paper, we propose a Bayesian clustered regression for spatially dependent data in order to detect clusters in the covariate effects. Our proposed method is based on the Dirichlet process which provides a probabilistic framework for simultaneous inference of the number of clusters and the clustering configurations. The usage of our method is illustrated both in simulation studies and an application to a housing cost dat…
Bayesian Additive Regression Trees (BART) is a flexible machine learning algorithm capable of capturing nonlinearities between an outcome and covariates and interaction among covariates. We extend BART to a semiparametric regression framework in which the conditional expectation of an outcome is a function of treatment, its effect modifiers, and confounders. The confounders, not of scientific interest, are allowed to have unspecified functional form, while treatment and other covariates that do have scientific importance are given the usual linear form from parametric regression. The result is…
Two key challenges in modern statistical applications are the large amount of information recorded per individual, and that such data are often not collected all at once but in batches. These batch effects can be complex, causing distortions in both mean and variance. We propose a novel sparse latent factor regression model to integrate such heterogeneous data. The model provides a tool for data exploration via dimensionality reduction while correcting for a range of batch effects. We study the use of several sparse priors (local and non-local) to learn the dimension of the latent factors. Our…
Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；University of Technology Sydney（悉尼科技大学）；Shanghai Jiao Tong University（上海交通大学）；JD.com Inc（京东集团）；JD Explore Academy（京东探索研究院）
Studies of the effects of medical interventions increasingly take place in distributed research settings using data from multiple clinical data sources including electronic health records and administrative claims. In such settings, privacy concerns typically prohibit sharing of individual patient data, and instead, analyses can only utilize summary statistics from the individual databases. In the specific but very common context of the Cox proportional hazards model, we show that standard meta analysis methods then lead to substantial bias when outcome counts are small. This bias derives prim…
In compositional data, detecting which part of the whole delineates heterogeneity is important. The aim is to propose a procedure to quantify this term in the multivariate regression context without abandoning the data's natural restriction. A single probabilistic model with a hierarchical structure was built for multiple compositional data. An objective criterion based on skewness and kurtosis metrics provides support to characterize each component's performance as well as to assist in choosing one component as a reference avoiding model identifiability issues. The inference procedure was don…
This paper studies the non-asymptotic merits of the double $\ell_1$-regularized for heterogeneous overdispersed count data via negative binomial regressions. Under the restricted eigenvalue conditions, we prove the oracle inequalities for Lasso estimators of two partial regression coefficients for the first time, using concentration inequalities of empirical processes. Furthermore, derived from the oracle inequalities, the consistency and convergence rate for the estimators are the theoretical guarantees for further statistical inference. Finally, both simulations and a real data analysis demo…
This paper studies the estimation of the coefficient matrix $\Ttheta$ in multivariate regression with hidden variables, $Y = (\Ttheta)^TX + (B^*)^TZ + E$, where $Y$ is a $m$-dimensional response vector, $X$ is a $p$-dimensional vector of observable features, $Z$ represents a $K$-dimensional vector of unobserved hidden variables, possibly correlated with $X$, and $E$ is an independent error. The number of hidden variables $K$ is unknown and both $m$ and $p$ are allowed but not required to grow with the sample size $n$. Since only $Y$ and $X$ are observable, we provide necessary conditions for t…
Optimum design via I-divergence for stable estimation in generalized regression models
广义回归模型中参数估计的最优设计是基于Fisher信息矩阵的标准化设计（参见Atkinson et al（2014）最近的一次阐述）。相应的最优性准则与这类模型中极大似然（ML）估计的渐近性质有关。然而，在有限样本试验中，ML估计的可辨识性、稳定性和唯一性可能存在问题，而这些问题并不反映在信息矩阵中。在Pázman和Pronzato（2014）中讨论了如何在标准非线性回归的实验设计阶段解决这些可估计性问题。在这里，我们希望将这种设计方法扩展到基于指数分布族的更一般的模型（二项分布、泊松分布、具有参数化方差的正态分布等）。其主要工具是信息（或Kullback-Leibler）散度，它与ML估计密切相关。
Recently, high-dimensional heterogeneous data have attracted a lot of attention and discussion. Under heterogeneity, semiparametric regression is a popular choice to model data in statistics. In this paper, we take advantages of expectile regression in computation and analysis of heterogeneity, and propose the regularized partially linear additive expectile regression with nonconvex penalty, for example, SCAD or MCP for such high-dimensional heterogeneous data. We focus on a more realistic scenario: the regression error is heavy-tailed distributed and only has finite moments, which is violated…
The paper continues the authors' work on the adaptive Wynn algorithm in a nonlinear regression model. In the present paper it is shown that if the mean response function satisfies a condition of `saturated identifiability', which was introduced by Pronzato \cite{Pronzato}, then the adaptive least squares estimators are strongly consistent. The condition states that the regression parameter is identifiable under any saturated design, i.e., the values of the mean response function at any $p$ distinct design points determine the parameter point uniquely where, typically, $p$ is the dimension of t…
Flexible estimation of heterogeneous treatment effects lies at the heart of many statistical challenges, such as personalized medicine and optimal resource allocation. In this paper, we develop a general class of two-step algorithms for heterogeneous treatment effect estimation in observational studies. We first estimate marginal effects and treatment propensities in order to form an objective function that isolates the causal component of the signal. Then, we optimize this data-adaptive objective function. Our approach has several advantages over existing methods. From a practical perspective…
MIT Economics（麻省理工学院经济学系）；Gatsby Unit, UCL（伦敦大学学院盖茨比研究组）
We propose estimators based on kernel ridge regression for nonparametric causal functions such as dose, heterogeneous, and incremental response curves. Treatment and covariates may be discrete or continuous in general spaces. Due to a decomposition property specific to the RKHS, our estimators have simple closed form solutions. We prove uniform consistency with finite sample rates via original analysis of generalized kernel ridge regression. We extend our main results to counterfactual distributions and to causal functions identified by front and back door criteria. We achieve state-of-the-art…
