In this paper, we study the classical problem of estimating the proportion of a finite population. First, we consider a fixed sample size method and derive an explicit sample size formula which ensures a mixed criterion of absolute and relative errors. Second, we consider an inverse sampling scheme such that the sampling is continue until the number of units having a certain attribute reaches a threshold value or the whole population is examined. We have established a simple method to determine the threshold so that a prescribed relative precision is guaranteed. Finally, we develop a multistage sampling scheme for constructing fixed-width confidence interval for the proportion of a finite population. Powerful computational techniques are introduced to make it possible that the fixed-width confidence interval ensures prescribed level of coverage probability.
In this paper we examine the properties of the Wilson score interval, used for inferences for an unknown binomial proportion parameter. We examine monotonicity and consistency properties of the interval and we generalise it to give two alternative forms for inferences undertaken in a finite population. We discuss the nature of the "finite population correction" in these generalised intervals and examine their monotonicity and consistency properties. This analysis gives the appropriate confidence interval for an unknown population proportion or unknown unsampled proportion in a finite or infini…
Cox proportional hazards model with measurement error is investigated. In Kukush et al. (2011) [Journal of Statistical Research 45, 77-94] and Chimisov and Kukush (2014) [Modern Stochastics: Theory and Applications 1, 13-32] asymptotic properties of simultaneous estimator $λ_n(\cdot)$, $β_n$ were studied for baseline hazard rate $λ(\cdot)$ and regression parameter $β$, at that the parameter set $Θ=Θ_λ\times Θ_β$ was assumed bounded. In the present paper, the set $Θ_λ$ is unbounded from above and not separated away from $0$. We construct the estimator in two steps: first we derive a strongly co…
Positive--unlabeled (PU) learning considers two samples, a positive set P with observations from only one class and an unlabeled set U with observations from two classes. The goal is to classify observations in U. Class mixture proportion estimation (MPE) in U is a key step in PU learning. Blanchard et al. [2010] showed that MPE in PU learning is a generalization of the problem of estimating the proportion of true null hypotheses in multiple testing problems. Motivated by this idea, we propose reducing the problem to one dimension via construction of a probabilistic classifier trained on the P…
In this paper, we propose a maximum smoothed likelihood method to estimate the component density functions of mixture models, in which the mixing proportions are known and may differ among observations. The proposed estimates maximize a smoothed log likelihood function and inherit all the important properties of probability density functions. A majorization-minimization algorithm is suggested to compute the proposed estimates numerically. In theory, we show that starting from any initial value, this algorithm increases the smoothed likelihood function and further leads to estimates that maximi…
Many insurance premium principles are defined and various estimation procedures introduced in the literature. In this paper, we focus on the estimation of the excess-of-loss reinsurance premium when the risks are randomly right-censored. The asymptotic normality of the proposed estimator is established under suitable conditions and its performance evaluated through sets of simulated data.
Thomas' partial likelihood estimator of regression parameters is widely used in the analysis of nested case-control data with Cox's model. This paper proposes a new estimator of the regression parameters, which is consistent and asymptotically normal. Its asymptotic variance is smaller than that of Thomas' estimator away from the null. Unlike some other existing estimators, the proposed estimator does not rely on any more data than strictly necessary for Thomas' estimator and is easily computable from a closed form estimating equation with a unique solution. The variance estimation is obtained…
Severe Ionosphere magnetic storms are feared events for integrity and continuity of navigation systems such as EGNOS, the European SBAS (Satellite-Based Augmentation System) complementing GPS and an accurate modelling of this event probability is necessary. Our aim for the work presented in this paper is to give an estimation of the frequency of such extreme magnetic storms per time unit (year) throughout a solar cycle. Thus, we develop an innovative approach based on a proportional hazard model, inspired by the Cox model, with time dependent covariates. The number of storms during a cycle is …
In the multiple testing context, a challenging problem is the estimation of the proportion $π_0$ of true-null hypotheses. A large number of estimators of this quantity rely on identifiability assumptions that either appear to be violated on real data, or may be at least relaxed. Under independence, we propose an estimator $\hatπ_0$ based on density estimation using both histograms and cross-validation. Due to the strong connection between the false discovery rate (FDR) and $π_0$, many multiple testing procedures (MTP) designed to control the FDR may be improved by introducing an estimator of $…
We introduce the problem of estimation of the parameters of a dynamically selected population in an infinite sequence of random variables and provide its application in the statistical inference based on record values from a non-stationary scheme. We develop unbiased estimation of the parameters of the dynamically selected population and evaluate the risk of the estimators. We provide comparisons with natural estimators and obtain asymptotic results. Finally, we illustrate the applicability of the results using real data.
The stratified proportional intensity model generalizes Cox's proportional intensity model by allowing different groups of the population under study to have distinct baseline intensity functions. In this article, we consider the problem of estimation in this model when the variable indicating the stratum is unobserved for some individuals in the studied sample. In this setting, we construct nonparametric maximum likelihood estimators for the parameters of the stratified model and we establish their consistency and asymptotic normality. Consistent estimators for the limiting variances are also…
When estimating a proportion and only a sample of triplets is given, dependencies within the triplets are to be accounted for. Without assuming a distribution for the success count of the triplet, together with the proportion, as second and third parameter the correlations of 1st and 2nd order enter the model. We apply maximum likelihood estimation, and derive consistency by using that the triplet count is multinomially distributed, combined with the continuous mapping theorem. The asymptotic normality follows with the delta-method, resulting in closed-form expressions for the standard errors.…
Given $n$ samples from a population of individuals belonging to different types with unknown proportions, how do we estimate the probability of discovering a new type at the $(n+1)$-th draw? This is a classical problem in statistics, commonly referred to as the missing mass estimation problem. Recent results by Ohannessian and Dahleh \citet{Oha12} and Mossel and Ohannessian \citet{Mos15} showed: i) the impossibility of estimating (learning) the missing mass without imposing further structural assumptions on the type proportions; ii) the consistency of the Good-Turing estimator for the missing …
Statistics Netherlands（荷兰统计局）；Radboud University（拉德堡德大学）；Eindhoven Univ. of Technology（埃因霍温理工大学）
Binary classifiers trained on a certain proportion of positive items introduce a bias when applied to data sets with different proportions of positive items. Most solutions for dealing with this issue assume that some information on the latter distribution is known. However, this is not always the case, certainly when this proportion is the target variable. In this paper a maximum likelihood estimator for the true proportion of positives in data sets is suggested and tested on synthetic and real world data.
We consider a binary branching process structured by a stochastic trait that evolves according to a diffusion process that triggers the branching events, in the spirit of Kimmel's model of cell division with parasite infection. Based on the observation of the trait at birth of the first n generations of the process, we construct nonparametric estimator of the transition of the associated bifurcating chain and study the parametric estimation of the branching rate. In the limit, as n tends to infinity, we obtain asymptotic efficiency in the parametric case and minimax optimality in the nonparame…
Interval estimation of the probability of success in a Binomial model is considered. Zieliński (2018) showed that the confidence interval which uses information about non-homogeneity of the sample is better than the classical one. In the following paper the shortest confidence interval for non-homogenous sample is constructed.
Large-scale multiple testing with highly correlated test statistics arises frequently in many scientific research. Incorporating correlation information in estimating false discovery proportion has attracted increasing attention in recent years. When the covariance matrix of test statistics is known, Fan, Han & Gu (2012) provided a consistent estimate of False Discovery Proportion (FDP) under arbitrary dependence structure. However, the covariance matrix is often unknown in many applications and such dependence information has to be estimated before estimating FDP (Efron, 2010). The estimation…
This paper considers a wide family of semiparametric repeated measures regression models, in which the main interest is on estimating population-level quantities such as mean, variance, probabilities etc. Examples of our framework include generalized linear models for clustered/longitudinal data, among many others. We derive plug-in kernel-based estimators of the population level quantities and derive their asymptotic distribution. An example involving estimation of the survival function of hemoglobin measures in the Kenya hemoglobin study data is presented to demonstrate our methodology.
We consider multiple testing means of many dependent Normal random variables that do not necessarily follow a joint Normal distribution. Under weak dependence, we show the uniform consistency of proportion estimators that are constructed as solutions to Lebesgue-Stieltjes equations for the setting of a point, bounded and one-sided null, respectively, and characterize via the index of weak dependence the sparsest proportion these estimators can consistently estimate. On the other hand, under a principal correlation structure and employing a suitable definition of p-value for composite null hypo…
This article arises as an alternative solution to the problem of estimating the future population of some care centers for children and adolescents at social risk. The population of these centers fluctuates from month to month due to various social, economic and political factors, so there is no way to calculate exactly how many children will enter and how many will leave. To make the estimate, a mathematical model was developed to estimate the population of the centers in the following month based on linear regressions and it was implemented in a spreadsheet to facilitate its use.
To better understand the interplay of censoring and sparsity we develop finite sample properties of nonparametric Cox proportional hazard's model. Due to high impact of sequencing data, carrying genetic information of each individual, we work with over-parametrized problem and propose general class of group penalties suitable for sparse structured variable selection and estimation. Novel non-asymptotic sandwich bounds for the partial likelihood are developed. We establish how they extend notion of local asymptotic normality (LAN) of Le Cam's. Such non-asymptotic LAN principles are further exte…
We consider a two-component mixture model with one known component. We develop methods for estimating the mixing proportion and the unknown distribution nonparametrically, given i.i.d.~data from the mixture model, using ideas from shape restricted function estimation. We establish the consistency of our estimators. We find the rate of convergence and asymptotic limit of the estimator for the mixing proportion. Completely automated distribution-free honest finite sample lower confidence bounds are developed for the mixing proportion. Connection to the problem of multiple testing is discussed. T…
Maximum likelihood estimation of mixture proportions has a long history, and continues to play an important role in modern statistics, including in development of nonparametric empirical Bayes methods. Maximum likelihood of mixture proportions has traditionally been solved using the expectation maximization (EM) algorithm, but recent work by Koenker & Mizera shows that modern convex optimization techniques -- in particular, interior point methods -- are substantially faster and more accurate than EM. Here, we develop a new solution based on sequential quadratic programming (SQP). It is substan…
Mixture-based Nonparametric Estimation of Spatial Covariance Functions with Applications to HIV Key Population Size Estimation across Sub-Saharan Africa
Capture-recapture methods aim to estimate the size of a closed population on the basis of multiple incomplete enumerations of individuals. In many applications, the individual probability of being recorded is heterogeneous in the population. Previous studies have suggested that it is not possible to reliably estimate the total population size when capture heterogeneity exists. Here we approach population estimation in the presence of capture heterogeneity as a latent length biased nonparametric density estimation problem on the unit interval. We show that in this setting it is generally imposs…
The proportional hazards assumption in the commonly used Cox model for censored failure time data is often violated in scientific studies. Yang and Prentice (2005) proposed a novel semiparametric two-sample model that includes the proportional hazards model and the proportional odds model as sub-models, and accommodates crossing survival curves. The model leaves the baseline hazard unspecified and the two model parameters can be interpreted as the short-term and long-term hazard ratios. Inference procedures were developed based on a pseudo score approach. Although extension to accommodate cova…
We develop here several goodness-of-fit tests for testing the k-monotonicity of a discrete density, based on the empirical distribution of the observations. Our tests are non-parametric, easy to implement and are proved to be asymptotically of the desired level and consistent. We propose an estimator of the degree of k-monotonicity of the distribution based on the non-parametric goodness-of-fit tests. We apply our work to the estimation of the total number of classes in a population. A large simulation study allows to assess the performances of our procedures.
We consider interval estimation of the difference between two binomial proportions. Several methods of constructing such an interval are known. Unfortunately those confidence intervals have poor coverage probability: it is significantly smaller than the nominal confidence level. In this paper a new confidence interval is proposed. The construction needs only information on sample sizes and sample difference between proportions. The coverage probability of the proposed confidence interval is at least the nominal confidence level. The new confidence interval is illustrated by a medical example.
The estimation of the potential impact fraction (including the population attributable fraction) with continuous exposure data frequently relies on strong distributional assumptions. However, these assumptions are often violated if the underlying exposure distribution is unknown or if the same distribution is assumed across time or space. Nonparametric methods to estimate the potential impact fraction are available for cohort data, but no alternatives exist for cross-sectional data. In this article, we discuss the impact of distributional assumptions in the estimation of the population impact …
In this paper, we develop an exact method for the determination of the minimum sample size for estimating the proportion of a finite population with prescribed margin of error and confidence level. By characterizing the behavior of the coverage probability with respect to the proportion, we show that the computational complexity can be significantly reduced and bounded regardless population size.
The problem of estimation of the proportion of units with a given attribute in a~finite population is considered. From the population a sample is drawn due to the simple random sampling without replacement. There are limited funds for conducting survey sample. Suppose that the population is divided into two strata. The question now arises: how should sample sizes be chosen from each strata to obtain the best estimation of proportion without exceeding the budget planned. In the paper it is shown, that with the appropriate sample allocation the variance of the stratified estimator may be reduced…
The paper is focused on the problem of estimating the probability $p$ of individual contaminated sample, under group testing. The precision of the estimator is given by the probability of proportional closeness, a concept defined in the Introduction. Two-stage and sequential sampling procedures are characterized. An adaptive procedure is examined.
We present an elementary mathematical method to find the minimax estimator of the Bernoulli proportion $θ$ under the squared error loss when $θ$ belongs to the restricted parameter space of the form $Ω= [0, η]$ for some pre-specified constant $0 \leq η\leq 1$. This problem is inspired from the problem of estimating the rate of positive COVID-19 tests. The presented results and applications would be useful materials for both instructors and students when teaching point estimation in statistical or machine learning courses.
We explore the Cauchy and a new heavy tailed (Fuquene, Perez and Pericchi (2011)) priors to estimate proportions on small areas. Hierarchical models and the Binomial likelihood in the exponential family form are used. We believe that the heavy tailed priors in survey sampling settings could be more effective than the choice of noninformative priors to eliminate antipathy towards methods that involve subjective elements or assumptions. To illustrate the robust Bayesian approach, we apply this methodology in a popular example: "the clement problem". Finally, we recommend to use the Cauchy prior …
The present paper presents the detail discussion on estimation of population mean in simple random sampling in the presence of non-response. Motivated by Gupta and Shabbir (2008), we have suggested the class of estimators of population mean using an auxiliary variable under non-response. A theoretical study is carried out using two-phase sampling scheme when the population mean of auxiliary variable is not known. An empirical study has also been done in the support of theoretical results.
The paper proposes some robust estimators of the finite population mean. Such estimators are particularly suitable in the presence of some outlying observations. Included as special cases of our general result are robust versions of the ratio estimator and the Horvitz-Thompson estimator. The robust estimators are derived on the basis of certain predictive influence functions.
An important issue raised by Efron in the context of large-scale multiple comparisons is that in many applications the usual assumption that the null distribution is known is incorrect, and seemingly negligible differences in the null may result in large differences in subsequent studies. This suggests that a careful study of estimation of the null is indispensable. In this paper, we consider the problem of estimating a null normal distribution, and a closely related problem, estimation of the proportion of non-null effects. We develop an approach based on the empirical characteristic function…
