In this paper, we develop an exact method for the determination of the minimum sample size for estimating the proportion of a finite population with prescribed margin of error and confidence level. By characterizing the behavior of the coverage probability with respect to the proportion, we show that the computational complexity can be significantly reduced and bounded regardless population size.
We model two time and space scales discrete observations by using a unique continuous diffusion process with time dependent coefficient. We define new parameters for the large scale model as functions of the small scale distribution cumulants. We use the non - uniform distribution of the observation time intervals to obtain consistent and unbiased estimators for these parameters. Closed form expressions for migration proportions between spatial domains are derived as functions of these parameters. The models are applied to estimate migration patterns from satellite tag data.
Estimating the proportion of signals hidden in a large amount of noise variables is of interest in many scientific inquires. In this paper, we consider realistic but theoretically challenging settings with arbitrary covariance dependence between variables. We define mean absolute correlation (MAC) to measure the overall dependence level and investigate a family of estimators for their performances in the full range of MAC. We explicit the joint effect of MAC dependence and signal sparsity on the performances of the family of estimators and discover that no single estimator in the family is mos…
We consider the problem of estimating the number of false null hypotheses among a very large number of independently tested hypotheses, focusing on the situation in which the proportion of false null hypotheses is very small. We propose a family of methods for establishing lower $100(1-α)%$ confidence bounds for this proportion, based on the empirical distribution of the $p$-values of the tests. Methods in this family are then compared in terms of ability to consistently estimate the proportion by letting $α\to 0$ as the number of hypothesis tests increases and the proportion decreases. This w…
An important issue raised by Efron in the context of large-scale multiple comparisons is that in many applications the usual assumption that the null distribution is known is incorrect, and seemingly negligible differences in the null may result in large differences in subsequent studies. This suggests that a careful study of estimation of the null is indispensable. In this paper, we consider the problem of estimating a null normal distribution, and a closely related problem, estimation of the proportion of non-null effects. We develop an approach based on the empirical characteristic function…
Multiple hypothesis testing is a fundamental problem in high dimensional inference, with wide applications in many scientific fields. In genome-wide association studies, tens of thousands of tests are performed simultaneously to find if any SNPs are associated with some traits and those tests are correlated. When test statistics are correlated, false discovery control becomes very challenging under arbitrary dependence. In the current paper, we propose a novel method based on principal factor approximation, which successfully subtracts the common dependence and weakens significantly the correl…
We introduce one-sided versions of Huber's contamination model, in which corrupted samples tend to take larger values than uncorrupted ones. Two intertwined problems are addressed: estimation of the mean of uncorrupted samples (minimum effect) and selection of corrupted samples (outliers). Regarding the minimum effect estimation, we derive the minimax risks and introduce adaptive estimators to the unknown number of contaminations. Interestingly, the optimal convergence rate highly differs from that in classical Huber's contamination model. Also, our analysis uncovers the effect of particular s…
A new formulation for the proportion of true null hypotheses $(π_0)$, based on the sum of all $p$-values and the average of expected $p$-value under the false null hypotheses has been proposed in the current work. This formulation of the parameter of interest $π_0$ has also been used to construct a new estimator for the same. The proposed estimator removes the problem of choosing tuning parameters in the existing estimators. Though the formulation is quite general, computation of the new estimator demands use of an initial estimate of $π_0$. The issue of choosing an appropriate initial estimat…
An important estimation problem that is closely related to large-scale multiple testing is that of estimating the null density and the proportion of nonnull effects. A few estimators have been introduced in the literature; however, several important problems, including the evaluation of the minimax rate of convergence and the construction of rate-optimal estimators, remain open. In this paper, we consider optimal estimation of the null density and the proportion of nonnull effects. Both minimax lower and upper bounds are derived. The lower bound is established by a two-point testing argument, …
In a recent paper (Efron (2004)), Efron pointed out that an important issue in large-scale multiple hypothesis testing is that the null distribution may be unknown and need to be estimated. Consider a Gaussian mixture model, where the null distribution is known to be normal but both null parameters--the mean and the variance--are unknown. We address the problem with a method based on Fourier transformation. The Fourier approach was first studied by Jin and Cai (2007), which focuses on the scenario where any non-null effect has either the same or a larger variance than that of the null effects.…
The paper is focused on the problem of estimating the probability $p$ of individual contaminated sample, under group testing. The precision of the estimator is given by the probability of proportional closeness, a concept defined in the Introduction. Two-stage and sequential sampling procedures are characterized. An adaptive procedure is examined.
In this paper, we develop an approach for the exact determination of the minimum sample size for estimating the parameter of an integer-valued random variable, which is parameterized by its expectation. Under some continuity and unimodal property assumptions, the exact computation is accomplished by reducing infinite many evaluations of coverage probability to finite many evaluations. Such a reduction is based on our discovery that the minimum of coverage probability with respect to the parameter bounded in an interval is attained at a discrete set of finite many values.
Suppose that we intend to perform an experiment consisting of a set of independent trials. The mean value of the response of each trial is assumed to be equal to the sum of the effect of the treatment selected for the trial, and some nuisance effects, e.g., the effect of a time trend, or blocking. In this model, we examine optimal approximate designs for the estimation of a system of treatment contrasts, with respect to a wide range of optimality criteria. We show that it is necessary for any optimal design to attain the optimal treatment proportions, which may be obtained from the marginal mo…
Consider the random Dirichlet partition of the interval into $n$ fragments with parameter $θ>0$. We recall the unordered Ewens sampling formulae from finite Dirichlet partitions. As this is a key variable for estimation purposes, focus is on the number of distinct visited species in the sampling process. These are illustrated in specific cases. We use these preliminary statistical results on frequencies distribution to address the following sampling problem: what is the estimated number of species when sampling is from Dirichlet populations? The obtained results are in accordance with the ones…
In this paper, we study the classical problem of estimating the proportion of a finite population. First, we consider a fixed sample size method and derive an explicit sample size formula which ensures a mixed criterion of absolute and relative errors. Second, we consider an inverse sampling scheme such that the sampling is continue until the number of units having a certain attribute reaches a threshold value or the whole population is examined. We have established a simple method to determine the threshold so that a prescribed relative precision is guaranteed. Finally, we develop a multistag…
Respondent-driven sampling (RDS) is a popular approach to study marginalized or hard-to-reach populations. It collects samples from a networked population by incentivizing participants to refer their friends into the study. One major challenge in analyzing RDS samples is seed bias. Seed bias refers to the fact that when the social network is divided into multiple communities (or blocks), the RDS sample might not provide a balanced representation of the different communities in the population, and such unbalance is correlated with the initial participant (or the seed). In this case, the distrib…
Estimating the number of unseen species is an important problem in many scientific endeavors. Its most popular formulation, introduced by Fisher, uses $n$ samples to predict the number $U$ of hitherto unseen species that would be observed if $t\cdot n$ new samples were collected. Of considerable interest is the largest ratio $t$ between the number of new and existing samples for which $U$ can be accurately predicted. In seminal works, Good and Toulmin constructed an intriguing estimator that predicts $U$ for all $t\le 1$, thereby showing that the number of species can be estimated for a popula…
