Mediation analyses are a statistical tool for testing the hypothesis about how the relationship between two variables may be direct or indirect via a third variable. Assessing statistical significance has been an area of active research; however, assessment of statistical power has been hampered by the lack of closed form calculations and the need for substantial amounts of computational simulations. The current work provides a detailed explanation of implementing large scale simulation procedures within a shared computing cluster environment. In addition, all results and code for implementing these procedures is publicly available. The resulting power analyses compare the effects of sample size and strength and direction of the relationships between the three variables. Comparisons of three confidence interval calculation methods demonstrated that the bias-corrected method is optimal and requires approximately ten less participants than the percentile method to achieve equivalent power. Differing strengths of distal and proximal effects were compared and did not differentially affect the power to detect mediation effects. Suppression effects were explored and demonstrate that in the presence of no observed relationship between two variables, entrance of the mediating variable into the model can reveal a suppressed relationship. The power to detect suppression effects is similar to unsuppressed mediation. These results and their methods provide important information about the power of mediation models for study planning. Of greater importance is that the methods lay the groundwork for assessment of statistical power of more complicated models involving multiple mediators and moderators.
Bootstrap inference is a powerful tool for obtaining robust inference for quantiles and difference-in-quantiles estimators. The computationally intensive nature of bootstrap inference has made it infeasible in large-scale experiments. In this paper, the theoretical properties of the Poisson bootstrap algorithm and quantile estimators are used to derive alternative resampling-free algorithms for Poisson bootstrap inference that reduce the computational complexity substantially without additional assumptions. These findings are connected to existing literature on analytical confidence intervals …
Linear mixed-effects models are commonly used to analyze clustered data structures. There are numerous packages to fit these models in R and conduct likelihood-based inference. The implementation of resampling-based procedures for inference are more limited. In this paper, we introduce the lmeresampler package for bootstrapping nested linear mixed-effects models fit via lme4 or nlme. Bootstrap estimation allows for bias correction, adjusted standard errors and confidence intervals for small samples sizes and when distributional assumptions break down. We will also illustrate how bootstrap resa…
We are concerned with nonparametric hypothesis testing of time series functionals. It is known that the popular autoregressive sieve bootstrap is, in general, not valid for statistics whose (asymptotic) distribution depends on moments of order higher than two, irrespective of whether the data come from a linear time series or a nonlinear one. Inspired by nonlinear system theory we circumvent this non-validity by introducing a higher-order bootstrap scheme based on the Volterra series representation of the process. In order to estimate coefficients of such a representation efficiently, we rely …
I have three goals in this article: (1) To show the enormous potential of bootstrapping and permutation tests to help students understand statistical concepts including sampling distributions, standard errors, bias, confidence intervals, null distributions, and P-values. (2) To dig deeper, understand why these methods work and when they don't, things to watch out for, and how to deal with these issues when teaching. (3) To change statistical practice---by comparing these methods to common t tests and intervals, we see how inaccurate the latter are; we confirm this with asymptotics. n >= 30 isn…
Over the last 35 years, several bootstrap methods for time series have been proposed. Popular `time-domain' methods include the block-bootstrap, the stationary bootstrap, the linear process bootstrap, etc.; subsampling for time series is also available, and is closely related to the block-bootstrap. `Frequency-domain' bootstrap has been performed either by resampling the periodogram ordinates or by resampling the ordinates of the Discrete Fourier Transform (DFT). The paper at hand proposes a novel construction of subsampling the DFT ordinates, and investigates its theoretical properties and re…
The asymptotic validity of a resampling method for two sequential processes constructed from non-degenerate $U$-statistics is established under mixing conditions. The resampling schemes, referred to as {\em dependent multiplier bootstraps}, result from an adaptation of the seminal approach of \cite{GomHor02} to mixing sequences. The proofs exploit recent results of \cite{DehWen10b} on degenerate $U$-statistics. A data-driven procedure for estimating a key bandwidth parameter involved in the resampling schemes is also suggested, making the use of the studied dependent multiplier bootstraps full…
The block bootstrap confidence interval based on dependent data can outperform the computationally more convenient normal approximation only with non-trivial Studentization which, in the case of complicated statistics, calls for highly specialist treatment. We propose two different approaches to improving the accuracy of the block bootstrap confidence interval under very general conditions. The first calibrates the coverage level by iterating the block bootstrap. The second calculates Studentizing factors directly from block bootstrap series and requires no non-trivial analytic treatment. Both…
Network models are applied in numerous domains where data can be represented as a system of interactions among pairs of actors. While both statistical and mechanistic network models are increasingly capable of capturing various dependencies amongst these actors, these dependencies imply the lack of independence. This poses statistical challenges for analyzing such data, especially when there is only a single observed network, and often leads to intractable likelihoods regardless of the modeling paradigm, which limit the application of existing statistical methods for networks. We explore a sub…
The technical details of the new bootstrap method of Shimodaira (2004) are given here in mathematical proofs as well as a supporting computer program. Approximately unbiased tests based on the bootstrap probabilities are considered in Shimodaira (2004) for the exponential family of distributions with unknown expectation parameter vector, where the null hypothesis is represented as an arbitrary-shaped region with smooth boundaries. It has been described in the lemmas of Shimodaira (2004) that the newly developed three-step multiscale bootstrap method calculates an asymptotically third-order acc…
Two key ingredients to carry out inference on the copula of multivariate observations are the empirical copula process and an appropriate resampling scheme for the latter. Among the existing techniques used for i.i.d. observations, the multiplier bootstrap of Rémillard and Scaillet (J. Multivariate Anal. 100 (2009) 377-386) frequently appears to lead to inference procedures with the best finite-sample properties. Bücher and Ruppert (J. Multivariate Anal. 116 (2013) 208-229) recently proposed an extension of this technique to strictly stationary strongly mixing observations by adapting the depe…
Bootstrap techniques (also called resampling computation techniques) have introduced new advances in modeling and model evaluation. Using resampling methods to construct a series of new samples which are based on the original data set, allows to estimate the stability of the parameters. Properties such as convergence and asymptotic normality can be checked for any particular observed data set. In most cases, the statistics computed on the generated data sets give a good idea of the confidence regions of the estimates. In this paper, we debate on the contribution of such methods for model selec…
Statistical resampling methods have become feasible for parametric estimation, hypothesis testing, and model validation now that the computer is a ubiquitous tool for statisticians. This essay focuses on the resampling technique for parametric estimation known as the Jackknife procedure. To outline the usefulness of the method and its place in the general class of statistical resampling techniques, I will quickly delineate two similar resampling methods: the bootstrap and the permutation test. I then outline the Jackknife method and show an example of its use.
An algorithm is described that enables efficient deterministic approximate computation of the bootstrap distribution for any linear bootstrap method $T_n^*$, alleviating the need for repeated resampling from observations (resp. input-derived data). In essence, the algorithm computes the distribution function from a linear mixture of independent random variables each having a finite discrete distribution. The algorithm is applicable to elementary bootstrap scenarios (targetting the mean as parameter of interest), for block bootstrap, as well as for certain residual bootstrap scenarios. Moreover…
With network data becoming ubiquitous in many applications, many models and algorithms for network analysis have been proposed. Yet methods for providing uncertainty estimates in addition to point estimates of network parameters are much less common. While bootstrap and other resampling procedures have been an effective general tool for estimating uncertainty from i.i.d. samples, adapting them to networks is highly nontrivial. In this work, we study three different network resampling procedures for uncertainty estimation, and propose a general algorithm to construct confidence intervals for ne…
The empirical beta copula is a simple but effective smoother of the empirical copula. Because it is a genuine copula, from which, moreover, it is particularly easy to sample, it is reasonable to expect that resampling procedures based on the empirical beta copula are expedient and accurate. In this paper, after reviewing the literature on some bootstrap approximations for the empirical copula process, we first show the asymptotic equivalence of several bootstrapped processes related to the empirical copula and empirical beta copula. Then we investigate the finite-sample properties of resamplin…
Approximately unbiased tests based on bootstrap probabilities are considered for the exponential family of distributions with unknown expectation parameter vector, where the null hypothesis is represented as an arbitrary-shaped region with smooth boundaries. This problem has been discussed previously in Efron and Tibshirani [Ann. Statist. 26 (1998) 1687-1718], and a corrected p-value with second-order asymptotic accuracy is calculated by the two-level bootstrap of Efron, Halloran and Holmes [Proc. Natl. Acad. Sci. U.S.A. 93 (1996) 13429-13434] based on the ABC bias correction of Efron [J. Amer…
The bootstrap is a versatile inference method that has proven powerful in many statistical problems. However, when applied to modern large-scale models, it could face substantial computation demand from repeated data resampling and model fitting. We present a bootstrap methodology that uses minimal computation, namely with a resample effort as low as one Monte Carlo replication, while maintaining desirable statistical guarantees. We present the theory of this method that uses a twisted perspective from the standard bootstrap principle. We also present generalizations of this method to nested s…
In this paper we consider the problem of bootstrapping a class of spatial regression models when the sampling sites are generated by a (possibly nonuniform) stochastic design and are irregularly spaced. It is shown that the natural extension of the existing block bootstrap methods for grid spatial data does not work for irregularly spaced spatial data under nonuniform stochastic designs. A variant of the blocking mechanism is proposed. It is shown that the proposed block bootstrap method provides a valid approximation to the distribution of a class of M-estimators of the spatial regression par…
Bootstrapping was designed to randomly resample data from a fixed sample using Monte Carlo techniques. However, the original sample itself defines a discrete distribution. Convolutional methods are well suited for discrete distributions, and we show the advantages of utilizing these techniques for bootstrapping. The discrete convolutional approach can provide exact numerical solutions for bootstrap quantities, or at least mathematical error bounds. In contrast, Monte Carlo bootstrap methods can only provide confidence intervals which converge slowly. Additionally, for some problems the computa…
The bootstrap, based on resampling, has, for several decades, been a widely used method for computing confidence intervals for applications where no exact method is available and when sample sizes are not large enough to be able to rely on easy-to-compute large-sample approximate methods, such a Wald (normal-approximation) confidence intervals. Simulation based bootstrap intervals have been proven useful in that their actual coverage probabilities are close to the nominal confidence level in small samples. Small samples analytical approximations such as the Wald method, however, tend to have c…
The process comparing the empirical cumulative distribution function of the sample with a parametric estimate of the cumulative distribution function is known as the empirical process with estimated parameters and has been extensively employed in the literature for goodness-of-fit testing. The simplest way to carry out such goodness-of-fit tests, especially in a multivariate setting, is to use a parametric bootstrap. Although very easy to implement, the parametric bootstrap can become very computationally expensive as the sample size, the number of parameters, or the dimension of the data incr…
Respondent-Driven Sampling (RDS) is a form of link-tracing sampling, a sampling technique used for `hard-to-reach' populations that aims to leverage individuals' social relationships to reach potential participants. While the methodological focus has been restricted to the estimation of population proportions, there is a growing interest in the estimation of uncertainty for RDS as recent findings suggest that most variance estimators underestimate variability. Recently, Baraff et al. (2016) proposed the \textit{tree bootstrap} method based on resampling the RDS recruitment tree, and empiricall…
Bootstrapping is a popular and computationally demanding resampling method used for measuring the accuracy of sample estimates and assisting with statistical inference. R is a freely available language and environment for statistical computing popular with biostatisticians for genomic data analyses. A survey of such R users highlighted its implementation of bootstrapping as a prime candidate for parallelization to overcome computational bottlenecks. The Simple Parallel R Interface (SPRINT) is a package that allows R users to exploit high performance computing in multi-core desktops and superco…
The two-sample problem for Cronbach's coefficient $α_C$, as an estimate of test or composite score reliability, has attracted little attention, compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistically how to compare two coefficients $α_{C,1}$ and $α_{C,2}$. The null hypothesis of interest is $H_0 : α_{C,1} = α_{C,2}$, which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation an…
Purdue University（普渡大学）；University of Missouri（密苏里大学）
In this paper, we propose a bootstrap method applied to massive data processed distributedly in a large number of machines. This new method is computationally efficient in that we bootstrap on the master machine without over-resampling, typically required by existing methods \cite{kleiner2014scalable,sengupta2016subsampled}, while provably achieving optimal statistical efficiency with minimal communication. Our method does not require repeatedly re-fitting the model but only applies multiplier bootstrap in the master machine on the gradients received from the worker machines. Simulations valid…
The bootstrap is a widely used procedure for statistical inference because of its simplicity and attractive statistical properties. However, the vanilla version of bootstrap is no longer feasible computationally for many modern massive datasets due to the need to repeatedly resample the entire data. Therefore, several improvements to the bootstrap method have been made in recent years, which assess the quality of estimators by subsampling the full dataset before resampling the subsamples. Naturally, the performance of these modern subsampling methods is influenced by tuning parameters such as …
With recent advances in high throughput technology, researchers often find themselves running a large number of hypothesis tests (thousands+) and esti- mating a large number of effect-sizes. Generally there is particular interest in those effects estimated to be most extreme. Unfortunately naive estimates of these effect-sizes (even after potentially accounting for multiplicity in a testing procedure) can be severely biased. In this manuscript we explore this bias from a frequentist perspective: we give a formal definition, and show that an oracle estimator using this bias dominates the naive …
We present a technique using data depth functions and resampling to perform best subset variable selection for a wide range of statistical models. We do this by assigning a score, called an $e$-value, to a candidate model, and use a fast bootstrap method to approximate sample versions of these $e$-values. Under general conditions, $e$-values can separate statistical models that adequately explain properties of the data from those that do not. This results in a fast algorithm that fits only a single model and evaluates $p +1$ models, $p$ being the number of predictors under consideration, as op…
It is common to show the confidence intervals or $p$-values of selected features, or predictor variables in regression, but they often involve selection bias. The selective inference approach solves this bias by conditioning on the selection event. Most existing studies of selective inference consider a specific algorithm, such as Lasso, for feature selection, and thus they have difficulties in handling more complicated algorithms. Moreover, existing studies often consider unnecessarily restrictive events, leading to over-conditioning and lower statistical power. Our novel and widely-applicabl…
A method using multiple imputation and bootstrap for dealing with missing data in mediation analysis is introduced and implemented in SAS. Through simulation studies, it is shown that the method performs well for both MCAR and MAR data without and with auxiliary variables. It is also shown that the method works equally well for MNAR data if auxiliary variables related to missingness are included. The application of the method is demonstrated through the analysis of a subset of data from the National Longitudinal Survey of Youth.
We develop and implement a novel fast bootstrap for dependent data. Our scheme is based on the i.i.d. resampling of the smoothed moment indicators. We characterize the class of parametric and semi-parametric estimation problems for which the method is valid. We show the asymptotic refinements of the proposed procedure, proving that it is higher-order correct under mild assumptions on the time series, the estimating functions, and the smoothing kernel. We illustrate the applicability and the advantages of our procedure for Generalized Empirical Likelihood estimation. As a by-product, our fast b…
We consider the performance of the bootstrap in high-dimensions for the setting of linear regression, where $p<n$ but $p/n$ is not close to zero. We consider ordinary least-squares as well as robust regression methods and adopt a minimalist performance requirement: can the bootstrap give us good confidence intervals for a single coordinate of $β$? (where $β$ is the true regression vector). We show through a mix of numerical and theoretical work that the bootstrap is fraught with problems. Both of the most commonly used methods of bootstrapping for regression -- residual bootstrap and pairs boo…
In a two-stage cluster sampling procedure, $n$ random populations are drawn independently from independent populations and a sub-sample of observations is taken in each of them. The estimator of the general mean of the observed variables is asymptotically Gaussian and the asymptotic distributions of several bootstrap versions of the normalized and studentized statistics are studied. A weighted population resampling provides a good approximation and its accuracy depends on the convergence rate of the sample size of the populations.
The construction of the simultaneous confidence bands for the integrated hazard function is considered. The Nelson--Aalen estimator is used. The simultaneous confidence bands based on bootstrap methods are presented. Two methods of construction of such confidence bands are proposed. The weird bootstrap method is used for resampling. Simulations are made to compare the actual coverage probability of the bootstrap and the asymptotic simultaneous confidence bands. It is shown that the equal--tailed bootstrap confidence band has the coverage probability closest to the nominal one. We also present …
We investigate the validity of two resampling techniques when carrying out inference on the underlying unknown copula using a recently proposed class of smooth, possibly data-adaptive nonparametric estimators that contains empirical Bernstein copulas (and thus the empirical beta copula). Following \cite{KirSegTsu21}, the first resampling technique is based on drawing samples from the smooth estimator and can only can be used in the case of independent observations. The second technique is a smooth extension of the so-called sequential dependent multiplier bootstrap and can thus be used in a ti…
We introduce a bootstrap procedure for high-frequency statistics of Brownian semistationary processes. More specifically, we focus on a hypothesis test on the roughness of sample paths of Brownian semistationary processes, which uses an estimator based on a ratio of realized power variations. Our new resampling method, the local fractional bootstrap, relies on simulating an auxiliary fractional Brownian motion that mimics the fine properties of high frequency differences of the Brownian semistationary process under the null hypothesis. We prove the first order validity of the bootstrap method …
A common question being raised in automatic speech recognition (ASR) evaluations is how reliable is an observed word error rate (WER) improvement comparing two ASR systems, where statistical hypothesis testing and confidence interval (CI) can be utilized to tell whether this improvement is real or only due to random chance. The bootstrap resampling method has been popular for such significance analysis which is intuitive and easy to use. However, this method fails in dealing with dependent data, which is prevalent in speech world - for example, ASR performance on utterances from the same speak…
