Variable selection is a difficult problem that is particularly challenging in the analysis of high-dimensional genomic data. Here, we introduce the CAR score, a novel and highly effective criterion for variable ranking in linear regression based on Mahalanobis-decorrelation of the explanatory variables. The CAR score provides a canonical ordering that encourages grouping of correlated predictors and down-weights antagonistic variables. It decomposes the proportion of variance explained and it is an intermediate between marginal correlation and the standardized regression coefficient. As a population quantity, any preferred inference scheme can be applied for its estimation. Using simulations we demonstrate that variable selection by CAR scores is very effective and yields prediction errors and true and false positive rates that compare favorably with modern regression techniques such as elastic net and boosting. We illustrate our approach by analyzing data concerned with diabetes progression and with the effect of aging on gene expression in the human brain. The R package "care" implementing CAR score regression is available from CRAN.
Background: The development of classification methods for personalized medicine is highly dependent on the identification of predictive genetic markers. In survival analysis it is often necessary to discriminate between influential and non-influential markers. Usually, the first step is to perform a univariate screening step that ranks the markers according to their associations with the outcome. It is common to perform screening using Cox scores, which quantify the associations between survival and each of the markers individually. Since Cox scores do not account for dependencies between the …
We proposed a new penalized method in this paper to solve sparse Poisson Regression problems. Being different from $\ell_1$ penalized log-likelihood estimation, our new method can be viewed as penalized weighted score function method. We show that under mild conditions, our estimator is $\ell_1$ consistent and the tuning parameter can be pre-specified, which shares the same good property of the square-root Lasso.
In recent years, there has been considerable theoretical development regarding variable selection consistency of penalized regression techniques, such as the lasso. However, there has been relatively little work on quantifying the uncertainty in these selection procedures. In this paper, we propose a new method for inference in high dimensions using a score test based on penalized regression. In this test, we perform penalized regression of an outcome on all but a single feature, and test for correlation of the residuals with the held-out feature. This procedure is applied to each feature in t…
The $\infty$-S test via regression quantile affine LASSO
University of Geneva（日内瓦大学）；Charles University Prague（布拉格查理大学）；National University of Defense and Technology（国防科技大学）；École Normale Supérieure Paris-Saclay（巴黎萨克雷高等师范学院）
Low-rank tensor decomposition generalizes low-rank matrix approximation and is a powerful technique for discovering low-dimensional structure in high-dimensional data. In this paper, we study Tucker decompositions and use tools from randomized numerical linear algebra called ridge leverage scores to accelerate the core tensor update step in the widely-used alternating least squares (ALS) algorithm. Updating the core tensor, a severe bottleneck in ALS, is a highly-structured ridge regression problem where the design matrix is a Kronecker product of the factor matrices. We show how to use approx…
From STLS to Projection-based Dictionary Selection in Sparse Regression for System Identification
AI Institute in Dynamic Systems, Department of Mechanical Engineering, University of Washington（动态系统人工智能研究所，机械工程系，华盛顿大学）；Department of Mechanical Engineering, University of Hawai‘i at Mānoa（机械工程系，夏威夷大学马诺阿分校）；Departamento de Matemática e Computação, Faculdade de Ciências e Tecnologia, Universidade Estadual Paulista “Júlio de Mesquita Filho”（数学与计算系，科技学院，保罗斯州立大学“朱利奥·梅斯奎塔·弗洛雷斯”分校）
We discuss an approach for fitting robust nonlinear regression models, which can be employed to model and predict the contagion dynamics of the Covid-19 in Italy. The focus is on the analysis of epidemic data using robust dose-response curves, but the functionality is applicable to arbitrary nonlinear regression models.
We propose a novel method for testing the null hypothesis of no effect of a covariate on the response in the context of functional linear concurrent regression. We establish an equivalent random effects formulation of our functional regression model under which our testing problem reduces to testing for zero variance component for random effects. For this purpose, we use a one-sided score test approach, which is an extension of the classical score test. We provide theoretical justification as to why our testing procedure has the right levels (asymptotically) under null using standard assumptio…
In other FICO Technical Papers, I have shown how to fit Generalized Additive Models (GAM) with shape constraints using quadratic programming applied to B-Spline component functions. In this paper, I extend the method to Robust Least Squares Regression.
In several FICO studies logistic regression has been shown to be a very competitive technology for developing unrestricted scoring models, especially for performance metrics like ROC area. Application of logistic regression has been hampered by the lack of software to handle complex score engineering such as shape and pattern constraints. The purpose of this paper is to develop a sequential quadratic programming algorithm for score engineered logistic regression. This approach is based on a simple Taylor series expansion of minus log likelihood, which is locally quadratic. and fits in with the…
In this paper we deal with the issue of performing accurate small-sample inference in the Birnbaum-Saunders regression model, which can be useful for modeling lifetime or reliability data. We derive a Bartlett-type correction for the score test and numerically compare the corrected test with the usual score test, the likelihood ratio test and its Bartlett-corrected version. Our simulation results suggest that the corrected test we propose is more reliable than the other tests.
In observational studies, researchers must select a method to control for confounding. Options include propensity score methods and regression. It remains unclear how dataset characteristics (size, overlap in propensity scores, exposure prevalence) influence the relative performance of the methods, making it difficult to select the best method for a particular dataset. A simulation study to evaluate the role of dataset characteristics on the performance of propensity score methods, compared to logistic regression, for estimating a marginal odds ratio in the presence of confounding was conducte…
Many schools in large urban districts have more applicants than seats. Centralized school assignment algorithms ration seats at over-subscribed schools using randomly assigned lottery numbers, non-lottery tie-breakers like test scores, or both. The New York City public high school match illustrates the latter, using test scores and other criteria to rank applicants at ``screened'' schools, combined with lottery tie-breaking at unscreened ``lottery'' schools. We show how to identify causal effects of school attendance in such settings. Our approach generalizes regression discontinuity methods t…
Explanations of the internal validity of regression discontinuity designs (RDD) generally appeal to the idea that RDDs are ``as good as" random near the treatment cut point. Cattaneo, Frandsen, and Titiunik (2015) are the first to take this justification to its full conclusion and propose estimating the RDD local average treatment effect (LATE) the same as one would a randomized experiment. This paper explores the implications of analyzing an RDD as a local random experiment when the running variable is a test score. I derive a formula for the bias in the LATE estimate estimated using the loca…
Many probabilistic models that have an intractable normalizing constant may be extended to contain covariates. Since the evaluation of the exact likelihood is difficult or even impossible for these models, score matching was proposed to avoid explicit computation of the normalizing constant. In the literature, score matching has so far only been developed for models in which the observations are independent and identically distributed (IID). However, the IID assumption does not hold in the traditional fixed design setting for regression-type models. To deal with the estimation of these covaria…
This paper presents a Bayesian approach to multiple-output quantile regression. The unconditional model is proven to be consistent and asymptotically correct frequentist confidence intervals can be obtained. The prior for the unconditional model can be elicited as the ex-ante knowledge of the distance of the tau-Tukey depth contour to the Tukey median, the first prior of its kind. A proposal for conditional regression is also presented. The model is applied to the Tennessee Project Steps to Achieving Resilience (STAR) experiment and it finds a joint increase in tau-quantile subpopulations for …
This work is motivated by a hand-collected data set from one of the largest Internet portals in Korea. This data set records the top 30 most frequently discussed stocks on its on-line message board. The frequencies are considered to measure the attention paid by investors to individual stocks. The empirical goal of the data analysis is to investigate the effect of this attention on trading behavior. For this purpose, we regress the (next day) returns and the (partially) observed ranks of frequencies. In the regression, the ranks are transformed into scores, for which purpose the identity or li…
Use copula to model dependency of variable extends multivariate gaussian assumption. In this paper we first empirically studied copula regression model with continous response. Both simulation study and real data study are given. Secondly we give a novel copula regression model with binary outcome, and we propose a score gradient estimation algorithms to fit the model. Both simulation study and real data study are given for our model and fitting algorithm.
In this paper, a practical estimation method for a regression model is proposed using semiparametric efficient score functions applicable to data with various shapes of errors. First, I derive semiparametric efficient score vectors for a homoscedastic regression model without any assumptions of errors. Next, the semiparametric efficient score function can be modified assuming a certain parametric distribution of errors according to the shape of the error distribution or by estimating the error distribution non-parametrically. Nonparametric methods for errors can be used to estimate the paramet…
Self and Mauritsen (1988) developed a sample size determination procedure for score tests in generalized linear models under contiguous alternatives. Its performance may deteriorate when the effect size is large. We propose a modification of the Self-Mauritsen method by taking into account of the variance of the score statistic under both the null and alternative hypotheses, and extend the method to noninferiority trials. The modified approach is employed to calculate the sample size for the logistic regression and negative binomial regression in superiority and noninferiority trials. We furth…
Multi-group data are commonly seen in practice. Such data structure consists of data from multiple groups and can be challenging to analyze due to data heterogeneity. We propose a novel Joint and Individual Component Regression (JICO) model to analyze multi-group data. In particular, our proposed model decomposes the response into shared and group-specific components, which are driven by low-rank approximations of joint and individual structures from the predictors respectively. The joint structure has the same regression coefficients across multiple groups, whereas individual structures have …
Data types that lie in metric spaces but not in vector spaces are difficult to use within the usual regression setting, either as the response and/or a predictor. We represent the information in these variables using distance matrices which requires only the specification of a distance function. A low-dimensional representation of such distance matrices can be obtained using methods such as multidimensional scaling. Once these variables have been represented as scores, an internal model linking the predictors and the response can be developed using standard methods. We call scoring the transfo…
Propensity score methods are common for estimating a binary treatment effect when treatment assignment is not randomized. When exposure is measured on an ordinal scale (i.e., low - medium - high), however, propensity score inference requires extensions which have received limited attention. Estimands of possible interest with an ordinal exposure are the average treatment effects between each pair of exposure levels. Using these estimands, it is possible to determine an optimal exposure level. Traditional methods, including dichotomization of the exposure or a series of binary propensity score …
Win-Ratio Regression for Prioritized Composite Outcomes in Observational Studies: Doubly Robust and Efficient Estimation with Future-Score Correction
University of Florida（佛罗里达大学）；University of California San Diego（加州大学圣地亚哥分校）；Stanford University School of Medicine（斯坦福大学医学院）；University of Rochester Medical Center（罗切斯特大学医学中心）
Prior proposals for cumulative statistics suggest making tiny random perturbations to the scores (independent variables in a regression) in order to ensure the scores' uniqueness. Uniqueness means that no score for any member of the population or subpopulation being analyzed is exactly equal to any other member's score. It turns out to be possible to construct from the original data a weighted data set that modifies the scores, weights, and responses (dependent variables in the regression) such that the new scores are unique and (together with the new weights and responses) yield the desired c…
High-dimensional data can be useful for causal inference by providing many confounders that may bolster the plausibility of the ignorability assumption. Propensity score methods are powerful tools for causal inference, are popular in health care research, and are particularly useful for high-dimensional data. Recent interest has surrounded a Bayesian formulation of these methods in order to flexibly estimate propensity scores and summarize posterior quantities while incorporating variance from the (potentially high-dimensional) treatment model. We discuss methods for Bayesian propensity score …
Propensity score weighting is widely used to improve the representativeness and correct the selection bias in the voluntary sample. The propensity score is often developed using a model for the sampling probability, which can be subject to model misspecification. In this paper, we consider an alternative approach of estimating the inverse of the propensity scores using the density ratio function satisfying the self-efficiency condition. The smoothed density ratio function is obtained by the solution to the information projection onto the space satisfying the moment conditions on the balancing …
Negative binomial regression is commonly employed to analyze overdispersed count data. With small to moderate sample sizes, the maximum likelihood estimator of the dispersion parameter may be subject to a significant bias, that in turn affects inference on mean parameters. This paper proposes inference for negative binomial regression based on adjustments of the score function aimed at mean and median bias reduction. The resulting estimating equations are similar to those available for improved inference in generalized linear models and, in particular, can be solved using a suitable extension …
Subsampling is a popular approach to alleviating the computational burden for analyzing massive datasets. Recent efforts have been devoted to various statistical models without explicit regularization. In this paper, we develop an efficient subsampling procedure for the large sample linear ridge regression. In contrast to the ordinary least square estimator, the introduction of the ridge penalty leads to a subtle trade-off between bias and variance. We first investigate the asymptotic properties of the subsampling estimator and then propose to minimize the asymptotic-mean-squared-error criteri…
Data analysis based on information from several sources is common in economic and biomedical studies. This setting is often referred to as the data fusion problem, which differs from traditional missing data problems since no complete data is observed for any subject. We consider a regression analysis when the outcome variable and some covariates are collected from two different sources. By leveraging the common variables observed in both data sets, doubly robust estimation procedures are proposed in the literature to protect against possible model misspecifications. However, they employ only …
On regression with estimated covariates and conditional effects given the propensity score
Rutgers, The State University of New Jersey（新泽西州立罗格斯大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Princeton University（普林斯顿大学）
It is often of interest to make inference on an unknown function that is a local parameter of the data-generating mechanism, such as a density or regression function. Such estimands can typically only be estimated at a slower-than-parametric rate in nonparametric and semiparametric models, and performing calibrated inference can be challenging. In many cases, these estimands can be expressed as the minimizer of a population risk functional. Here, we propose a general framework that leverages such representation and provides a nonparametric extension of the score test for inference on an infini…
Combining information from multiple samples is often needed in biomedical and economic studies, but the differences between these samples must be appropriately taken into account in the analysis of the combined data. We study estimation for moment restriction models with data combination from two samples under an ignorablility-type assumption but allowing for different marginal distributions of common variables between the two samples. Suppose that an outcome regression model and a propensity score model are specified. By leveraging the semiparametric efficiency theory, we derive an augmented …
Covariate Balancing Value Estimation for Optimal Individualized Treatment Rules
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；School of Mathematics and Statistics, Beijing Technology and Business University（北京工商大学数学与统计学院）
In a standard regression problem, we have a set of explanatory variables whose effect on some response vector is modeled. For wide binary data, such as genetic marker data, we often have two limitations. First, we have more parameters than observations. Second, main effects are not the main focus; instead the primary aim is to uncover interactions between the binary variables that effect the response. Methods such as logic regression are able to find combinations of the explanatory variables that capture higher-order relationships in the response. However, the number of explanatory variables t…
Tie-breaker experimental designs are hybrids of Randomized Controlled Trials (RCTs) and Regression Discontinuity Designs (RDDs) in which subjects with moderate scores are placed in an RCT while subjects with extreme scores are deterministically assigned to the treatment or control group. In settings where it is unfair or uneconomical to deny the treatment to the more deserving recipients, the tie-breaker design (TBD) trades off the practical advantages of the RDD with the statistical advantages of the RCT. The practical costs of the randomization in TBDs can be hard to quantify in generality, …
