Asymptotic Analysis of Sampling Estimators for Randomized Numerical Linear Algebra Algorithms
The University of Georgia（佐治亚大学）；University of California at San Diego（加利福尼亚大学圣地亚哥分校）；Harvard University（哈佛大学）；Central University of Finance and Economics（中央财经大学）；International Computer Science Institute（国际计算机科学研究所）；University of California at Berkeley（加利福尼亚大学伯克利分校）
Ping Ma, Xinlian Zhang, Xin Xing, Jingyi Ma, Michael W. Mahoney
The statistical analysis of Randomized Numerical Linear Algebra (RandNLA) algorithms within the past few years has mostly focused on their performance as point estimators. However, this is insufficient for conducting statistical inference, e.g., constructing confidence intervals and hypothesis testing, since the distribution of the estimator is lacking. In this article, we develop an asymptotic analysis to derive the distribution of RandNLA sampling estimators for the least-squares problem. In particular, we derive the asymptotic distribution of a general sampling estimator with arbitrary sampling probabilities. The analysis is conducted in two complementary settings, i.e., when the objective of interest is to approximate the full sample estimator or is to infer the underlying ground truth model parameters. For each setting, we show that the sampling estimator is asymptotically normally distributed under mild regularity conditions. Moreover, the sampling estimator is asymptotically unbiased in both settings. Based on our asymptotic analysis, we use two criteria, the Asymptotic Mean Squared Error (AMSE) and the Expected Asymptotic Mean Squared Error (EAMSE), to identify optimal sampling probabilities. Several of these optimal sampling probability distributions are new to the literature, e.g., the root leverage sampling estimator and the predictor length sampling estimator. Our theoretical results clarify the role of leverage in the sampling process, and our empirical results demonstrate improvements over existing methods.
This paper studies one-sided hypothesis testing under random sampling without replacement. That is, when $n+1$ binary random variables $X_1,\ldots, X_{n+1}$ are subject to a permutation invariant distribution and $n$ binary random variables $X_1,\ldots, X_{n}$ are observed, we have proposed randomized tests with a randomization parameter for the upper confidence limit of the expectation of the $(n+1)$th random variable $X_{n+1}$ under a given significance level $δ>0$. Our proposed randomized test significantly improves over deterministic test unlike random sampling with replacement.
University of California, Davis（加利福尼亚大学戴维斯分校）；University of Toronto（多伦多大学）；Vector Institute（矢量研究所）
The randomized midpoint method, proposed by [SL19], has emerged as an optimal discretization procedure for simulating the continuous time Langevin diffusions. Focusing on the case of strong-convex and smooth potentials, in this paper, we analyze several probabilistic properties of the randomized midpoint discretization method for both overdamped and underdamped Langevin diffusions. We first characterize the stationary distribution of the discrete chain obtained with constant step-size discretization and show that it is biased away from the target distribution. Notably, the step-size needs to g…
Let $f:[0,1)^d \to {\mathbb R}$ be an integrable function. An objective of many computer experiments is to estimate $\int_{[0,1)^d} f(x) dx$ by evaluating f at a finite number of points in [0,1)^d. There is a design issue in the choice of these points and a popular choice is via the use of randomized orthogonal arrays. This article proves a multivariate central limit theorem for a class of randomized orthogonal array sampling designs [Owen (1992a)] as well as for a class of OA-based Latin hypercubes [Tang (1993)].
We introduce a very general method for sparse and large-scale variable selection. The large-scale regression settings is such that both the number of parameters and the number of samples are extremely large. The proposed method is based on careful combination of penalized estimators, each applied to a random projection of the sample space into a low-dimensional space. In one special case that we study in detail, the random projections are divided into non-overlapping blocks; each consisting of only a small portion of the original data. Within each block we select the projection yielding the sm…
Two-stage randomized experiments are becoming an increasingly popular experimental design for causal inference when the outcome of one unit may be affected by the treatment assignments of other units in the same cluster. In this paper, we provide a methodological framework for general tools of statistical inference and power analysis for two-stage randomized experiments. Under the randomization-based framework, we consider the estimation of a new direct effect of interest as well as the average direct and spillover effects studied in the literature. We provide unbiased estimators of these caus…
The aim of this paper is to generalize the PAC-Bayesian theorems proved by Catoni in the classification setting to more general problems of statistical inference. We show how to control the deviations of the risk of randomized estimators. A particular attention is paid to randomized estimators drawn in a small neighborhood of classical estimators, whose study leads to control the risk of the latter. These results allow to bound the risk of very general estimation procedures, as well as to perform model selection.
Regression adjustments are often considered by investigators to improve the estimation efficiency of causal effect in randomized experiments when there exists many pre-experiment covariates. In this paper, we provide conditions that guarantee the penalized regression including the Ridge, Elastic Net and Adapive Lasso adjusted causal effect estimators are asymptotic normal and we show that their asymptotic variances are no greater than that of the simple difference-in-means estimator, as long as the penalized estimators are risk consistent. We also provide conservative estimators for the asympt…
We present a new estimator of the restricted mean survival time in randomized trials where there is right censoring that may depend on treatment and baseline variables. The proposed estimator leverages prognostic baseline variables to obtain equal or better asymptotic precision compared to traditional estimators. Under regularity conditions and random censoring within strata of treatment and baseline variables, the proposed estimator has the following features: (i) it is interpretable under violations of the proportional hazards assumption; (ii) it is consistent and at least as precise as the …
We provide a principled way for investigators to analyze randomized experiments when the number of covariates is large. Investigators often use linear multivariate regression to analyze randomized experiments instead of simply reporting the difference of means between treatment and control groups. Their aim is to reduce the variance of the estimated treatment effect by adjusting for covariates. If there are a large number of covariates relative to the number of observations, regression may perform poorly because of overfitting. In such cases, the Lasso may be helpful. We study the resulting La…
We propose a consistent estimator of sharp bounds on the variance of the difference-in-means estimator in completely randomized experiments. Generalizing Robins [Stat. Med. 7 (1988) 773-785], our results resolve a well-known identification problem in causal inference posed by Neyman [Statist. Sci. 5 (1990) 465-472. Reprint of the original 1923 paper]. A practical implication of our results is that the upper bound estimator facilitates the asymptotically narrowest conservative Wald-type confidence intervals, with applications in randomized controlled and clinical trials.
Randomized Midpoint Method for Log-Concave Sampling under Constraints
Department of Mathematical Sciences, Tsinghua University（清华大学数学科学系）；Artificial Intelligence Research Institute, Shenzhen MSU-BIT University（深圳MSU-BIT大学人工智能研究院）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）
We establish the inferential properties of the mean-difference estimator for the average treatment effect in randomized experiments where each unit in a population is randomized to one of two treatments and then units within treatment groups are randomly sampled. The properties of this estimator are well-understood in the experimental design scenario where first units are randomly sampled and then treatment is randomly assigned, but not for the aforementioned scenario where the sampling and treatment assignment stages are reversed. We find that the inferential properties of the mean-difference…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Peking University（北京大学）
Neural networks (NNs) are known to be vulnerable against adversarial perturbations, and thus there is a line of work aiming to provide robustness certification for NNs, such as randomized smoothing, which samples smoothing noises from a certain distribution to certify the robustness for a smoothed classifier. However, as shown by previous work, the certified robust radius in randomized smoothing suffers from scaling to large datasets ("curse of dimensionality"). To overcome this hurdle, we propose a Double Sampling Randomized Smoothing (DSRS) framework, which exploits the sampled probability f…
We study conditions under which treatment effect estimators constructed under the no-interference assumption in randomized experiments are asymptotically normal in the presence of interference. We prove that the standard Horvitz-Thompson estimator is asymptotically normal under a restricted interference condition characterized by limiting the degree of the dependency graph. The amount of interference is allowed to grow with the population size. We then provide a central limit theorem for the difference-in-means estimator that can handle interference that exists between all pairs of units, prov…
Researchers often use linear regression to analyse randomized experiments to improve treatment effect estimation by adjusting for imbalances of covariates in the treatment and control groups. Our work offers a randomization-based inference framework for regression adjustment in stratified randomized experiments. Under mild conditions, we re-establish the finite population central limit theorem for a stratified experiment. We prove that both the stratified difference-in-means and the regression-adjusted average treatment effect estimators are consistent and asymptotically normal. The asymptotic…
Graduate School of Business and Department of Economics, Stanford University（斯坦福大学商学院与经济系）；Applied Statistics Unit, Statistical Sciences Division, Indian Statistical Institute（印度统计研究所统计科学系应用统计单元）
This paper develops a flexible method for decreasing the variance of estimators for complex experiment effect metrics (e.g. ratio metrics) while retaining asymptotic unbiasedness. This method uses the auxiliary information about the experiment units to decrease the variance. The method can incorporate almost any arbitrary predictive model (e.g. linear regression, regularization, neural networks) to adjust the estimators. The adjustment involves some free parameters which can be optimized to achieve the smallest variance reduction given the predictive model performance. Also we approximate the …
In a randomized controlled trial, treatment switching (also called contamination or crossover) occurs when a patient initially assigned to one treatment arm changes to another arm during the course of follow-up. Overlooking treatment switching might substantially bias the evaluation of treatment efficacy or safety. To account for treatment switching, instrumental variable (IV) methods by leveraging the initial randomized assignment as an IV serve as natural adjustment methods because they allow dependent treatment switching possibly due to underlying prognoses. However, the ``exclusion restric…
In many epidemiological contexts, disease occurrences and their rates are naturally modelled by counting processes and their intensities, allowing an analysis based on martingale methods. These methods lend themselves to extensions of nested case-control sampling designs where general methods of control selection can be easily incorporated. This same methodology allows for extensions of the Mantel-Haenszel estimator in two main directions. First, a variety of new sampling designs can be incorporated which can yield substantial efficiency gains over simple random sampling. Second, the extension…
In this paper we have proposed a median based estimator using known value of some population parameter(s) in simple random sampling. Various existing estimators are shown particular members of the proposed estimator. The bias and mean squared error of the proposed estimator is obtained up to the first order of approximation under simple random sampling without replacement. An empirical study is carried out to judge the superiority of proposed estimator over others.
Minimization of a stochastic cost function is commonly used for approximate sampling in high-dimensional Bayesian inverse problems with Gaussian prior distributions and multimodal posterior distributions. The density of the samples generated by minimization is not the desired target density, unless the observation operator is linear, but the distribution of samples is useful as a proposal density for importance sampling or for Markov chain Monte Carlo methods. In this paper, we focus on applications to sampling from multimodal posterior distributions in high dimensions. We first show that samp…
Kadilar and Cingi (2003) have introduced a family of estimators using auxiliary information in stratified random sampling. In this paper, we propose the ratio estimator for the estimation of population mean in the stratified random sampling by using the estimators in Bahl and Tuteja (1991) and Kadilar and Cingi (2003). Obtaining the mean square error (MSE) equations of the proposed estimators, we find theoretical conditions that the proposed estimators are more efficient than the other estimators. These theoretical findings are supported by a numerical example.
In situations where the sampling units in a study can be more easily ranked based on the measurement of an auxiliary variable, ranked set sampling provide unbiased estimators for the mean of a population that they are more efficient than unbiased estimator based on simple random sample. In this paper, we consider the Morgenstern type bivariate generalized exponential distribution (MTBGED) and obtain several unbiased estimators for a parameter mean of the marginal distribution of MTBGED based on different ranked set sampling schemes. The efficiency of all considered estimators are evaluate and …
Singh et al (20009) introduced a family of exponential ratio and product type estimators in stratified random sampling. Under stratified random sampling without replacement scheme, the expressions of bias and mean square error (MSE) of Singh et al (2009) and some other estimators, up to the first- and second-order approximations are derived. Also, the theoretical findings are supported by a numerical example.
In this paper, a procedure is given for estimating the population mean in simple random sampling without replacement in the presence of auxiliary information. The mean squared error expressions of the proposed estimators have been derived under large sample approximation. We have compared the performances of the proposed estimators with several existing estimators. Both theoretical and empirical findings are encouraging and support the soundness of the proposed procedure for mean estimation.
Some improved estimators are proposed for estimating the population mean in stratified sampling in the presence of auxiliary information. Mean square error (MSE) of the proposed estimators have been derived under large sample approximation. It has been shown that under optimum conditions proposed estimators are better than usual unbiased estimator and Hansen (1946) estimator. Both theoretical and empirical findings are encouraging and support the soundness of the proposed procedure for mean estimation.
In a completely randomized experiment, the variances of treatment effect estimators in the finite population are usually not identifiable and hence not estimable. Although some estimable bounds of the variances have been established in the literature, few of them are derived in the presence of covariates. In this paper, the difference-in-means estimator and the Wald estimator are considered in the completely randomized experiment with perfect compliance and noncompliance, respectively. Sharp bounds for the variances of these two estimators are established when covariates are available. Further…
