We develop methods for estimating the size of hard-to-reach populations from data collected using network-based questions on standard surveys. Such data arise by asking respondents how many people they know in a specific group (e.g., people named Michael, intravenous drug users). The Network Scale up Method (NSUM) is a tool for producing population size estimates using these indirect measures of respondents' networks. Killworth et al. [Soc. Netw. 20 (1998a) 23-50, Evaluation Review 22 (1998b) 289-308] proposed maximum likelihood estimators of population size for a fixed effects model in which respondents' degrees or personal network sizes are treated as fixed. We extend this by treating personal network sizes as random effects, yielding principled statements of uncertainty. This allows us to generalize the model to account for variation in people's propensity to know people in particular subgroups (barrier effects), such as their tendency to know people like themselves, as well as their lack of awareness of or reluctance to acknowledge their contacts' group memberships (transmission bias). NSUM estimates also suffer from recall bias, in which respondents tend to underestimate the number of members of larger groups that they know, and conversely for smaller groups. We propose a data-driven adjustment method to deal with this. Our methods perform well in simulation studies, generating improved estimates and calibrated uncertainty intervals, as well as in back estimates of real sample data. We apply them to data from a study of HIV/AIDS prevalence in Curitiba, Brazil. Our results show that when transmission bias is present, external information about its likely extent can greatly improve the estimates. The methods are implemented in the NSUM R package.
In this paper, we propose a transformed naïve ratio and product based estimators using the characterizing scalar in presence of auxiliary information of the study variable for estimating the population mode following simple random sampling without replacement. The bias, mean square errors, relative efficiency, ratios of the exact values of mean square errors to the simulated mean square errors and confidence interval are studied for the performance of the proposed transformed naïve ratio type estimator with the certain natural population as well as artificially generated data sets. We have sho…
Capture-recapture methods aim to estimate the size of a closed population on the basis of multiple incomplete enumerations of individuals. In many applications, the individual probability of being recorded is heterogeneous in the population. Previous studies have suggested that it is not possible to reliably estimate the total population size when capture heterogeneity exists. Here we approach population estimation in the presence of capture heterogeneity as a latent length biased nonparametric density estimation problem on the unit interval. We show that in this setting it is generally imposs…
Given p independent normal populations, we consider the problem of estimating the mean of those populations, that based on the observed data, give the strongest signals. We explicitly condition on the ranking of the sample means, and consider a constrained conditional maximum likelihood (CCMLE) approach, avoiding the use of any priors and of any sparsity requirement between the population means. Our results show that if the observed means are too close together, we should in fact use the grand mean to estimate the mean of the population with the larger sample mean. If they are separated by mor…
In the near future, millions of load curves measuring the electricity consumption of French households in small time grids (probably half hours) will be available. All these collected load curves represent a huge amount of information which could be exploited using survey sampling techniques. In particular, the total consumption of a specific cus- tomer group (for example all the customers of an electricity supplier) could be estimated using unequal probability random sampling methods. Unfortunately, data collection may undergo technical problems resulting in missing values. In this paper we s…
The paper describes a new class of capture-recapture models for closed populations when individual covariates are available. The novelty consists in combining a latent class model for the distribution of the capture history, where the class weights and the conditional distributions given the latent may depend on covariates, with a model for the marginal distribution of the available covariates as in \cite{Liu2017}. In addition, any general form of serial dependence is allowed when modeling capture histories conditionally on the latent and covariates. A Fisher-scoring algorithm for maximum like…
For estimating area-specific parameters (quantities) in a finite population, a mixed model prediction approach is attractive. However, this approach strongly depends on the normality assumption of the response values although we often encounter a non-normal case in practice. In such a case, transforming observations to make them suitable for normality assumption is a useful tool, but the problem of selecting suitable transformation still remains open. To overcome the difficulty, we here propose a new empirical best predicting method by using a parametric family of transformations to estimate a…
Weighted estimators are commonly used for estimating exposure effects in observational settings to establish causal relations. These estimators have a long history of development when the exposure of interest is binary and where the weights are typically functions of an estimated propensity score. Recent developments in optimization-based estimators for constructing weights in binary exposure settings, such as those based on entropy balancing, have shown more promise in estimating treatment effects than those methods that focus on the direct estimation of the propensity score using likelihood-…
Estimation of population size using incomplete lists (also called the capture-recapture problem) has a long history across many biological and social sciences. For example, human rights and other groups often construct partial and overlapping lists of victims of armed conflicts, with the hope of using this information to estimate the total number of victims. Earlier statistical methods for this setup either use potentially restrictive parametric assumptions, or else rely on typically suboptimal plug-in-type nonparametric estimators; however, both approaches can lead to substantial bias, the fo…
New York University（纽约大学）；North Carolina State University（北卡罗来纳州立大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Reinforcement Learning (RL) has the promise of providing data-driven support for decision-making in a wide range of problems in healthcare, education, business, and other domains. Classical RL methods focus on the mean of the total return and, thus, may provide misleading results in the setting of the heterogeneous populations that commonly underlie large-scale datasets. We introduce the K-Heterogeneous Markov Decision Process (K-Hetero MDP) to address sequential decision problems with population heterogeneity. We propose the Auto-Clustered Policy Evaluation (ACPE) for estimating the value of …
In this paper we introduce the concept of bootstrapped pivots for the sample and the population means. This is in contrast to the classical method of constructing bootstrapped confidence intervals for the population mean via estimating the cutoff points via drawing a number of bootstrap sub-samples. We show that this new method leads to constructing asymptotic confidence intervals with significantly smaller error in comparison to both of the traditional t-intervals and the classical bootstrapped confidence intervals. The approach taken in this paper relates naturally to super-population modeli…
A new strategy is introduced for estimating population size and networked population characteristics. Sample selection is based on a multi-wave snowball sampling design. A generalized stochastic block model is posited for the population's network graph. Inference is based on a Bayesian data augmentation procedure. Applications are provided to an empirical and simulated populations. The results demonstrate that statistically efficient estimates of the size and distribution of the population can be achieved.
We present a new design and inference method for estimating population size of a hidden population best reached through a link-tracing design. The strategy involves the Rao-Blackwell Theorem applied to a sufficient statistic markedly different from the usual one that arises in sampling from a finite population. An empirical application is described. The result demonstrates that the strategy can efficiently incorporate adaptively selected members of the sample into the inference procedure.
Comment on ``Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data'' [arXiv:0804.2958]
Comment on ``Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data'' [arXiv:0804.2958]
Rejoinder to ``Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data'' [arXiv:0804.2958]
A new approach to estimate population size based on a stratified link-tracing sampling design is presented. The method extends on the Frank and Snijders (1994) approach by allowing for heterogeneity in the initial sample selection procedure. Rao-Blackwell estimators and corresponding resampling approximations similar to that detailed in Vincent and Thompson (2017) are explored. An empirical application is provided for a hard-to-reach networked population. The results demonstrate that the approach has much potential for application to such populations. Supplementary materials for this article a…
In this paper we consider the estimation of population size from one-source capture--recapture data, that is, a list in which individuals can potentially be found repeatedly and where the question is how many individuals are missed by the list. As a typical example, we provide data from a drug user study in Bangkok from 2001 where the list consists of drug users who repeatedly contact treatment institutions. Drug users with 1, 2, 3$,...$ contacts occur, but drug users with zero contacts are not present, requiring the size of this group to be estimated. Statistically, these data can be consider…
We read with interest the article by di Cecco et al. (2018), but have reservations about the usefulness of the latent class model specifically for estimating overcoverage. In particular, we question the interpretation of the parameters of the fitted latent class model.
Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
We study the problem of estimating the distribution of effect sizes (the mean of the test statistic under the alternate hypothesis) in a multiple testing setting. Knowing this distribution allows us to calculate the power (type II error) of any experimental design. We show that it is possible to estimate this distribution using an inexpensive pilot experiment, which takes significantly fewer samples than would be required by an experiment that identified the discoveries. Our estimator can be used to guarantee the number of discoveries that will be made using a given experimental design in a fu…
We propose a two-step estimating procedure for generalized additive partially linear models with clustered data using estimating equations. Our proposed method applies to the case that the number of observations per cluster is allowed to increase with the number of independent subjects. We establish oracle properties for the two-step estimator of each function component such that it performs as well as the univariate function estimator by assuming that the parametric vector and all other function components are known. Asymptotic distributions and consistency properties of the estimators are ob…
Ecologists are interested in modeling the population growth of species in various ecosystems. Studying population dynamics can assist environmental managers in making better decisions for the environment. Traditionally, the sampling of species and tracking of populations have been recorded on a regular time frequency. However, sampling can be an expensive process due to available resources, money and time. Limiting sampling makes it challenging to properly track the growth of a population. Thus, we propose a new and novel approach to designing sampling regimes based on the dynamics associated …
Paired estimation of change in parameters of interest over a population plays a central role in several application domains including those in the social sciences, epidemiology, medicine and biology. In these domains, the size of the population under study is often very large, however, the number of observations available per individual in the population is very small (\emph{sparse observations}) which makes the problem challenging. Consider the setting with $N$ independent individuals, each with unknown parameters $(p_i, q_i)$ drawn from some unknown distribution on $[0, 1]^2$. We observe $X_…
Estimating the unknown number of classes in a population has numerous important applications. In a Poisson mixture model, the problem is reduced to estimating the odds that a class is undetected in a sample. The discontinuity of the odds prevents the existence of locally unbiased and informative estimators and restricts confidence intervals to be one-sided. Confidence intervals for the number of classes are also necessarily one-sided. A sequence of lower bounds to the odds is developed and used to define pseudo maximum likelihood estimators for the number of classes.
In this paper, a new modification of ranked set sampling (RSS) is suggested, namely; unified ranked set sampling (URSS) for estimating the population mean and variance. The performance of the empirical mean and variance estimators based on URSS are compared with their counterparts in ranked set sampling and simple random sampling (SRS) via Monte Carlo simulation. Simulation results indicate that the URSS estimators perform better than their counterparts using RSS and SRS designs when the ranking is perfect. When the ranking is imperfect, the URSS estimators still are superior than their counte…
RCTs sometimes test interventions that aim to improve existing services targeted to a subset of individuals identified after randomization. Accordingly, the treatment could affect the composition of service recipients and the offered services. With such bias, intention-to-treat estimates using data on service recipients and nonrecipients may be difficult to interpret. This article develops causal estimands and inverse probability weighting (IPW) estimators for complier populations in these settings, using a generalized estimating equation approach that adjusts the standard errors for estimatio…
For Dual-record system, in the context of human population, the popular Chandrasekar-Deming model incorporates only the time variation effect on capture probabilities. How-ever, in practice population may undergo behavioral change after being captured first time. In this paper we focus on the Dual-record system model (equivalent to capture- recapture model with two sampling occasions) with both the time as well as behavioral response variation. The relevant model suffers from identifiability problem. Two approaches are proposed from which approximate Bayes estimates can be obtained using very …
We investigate the problem of estimating the product of means of independent populations from the one parameter exponential family in a Bayesian framework. We give a random design which allocates mi the number of observations from population Pi such that the Bayes risk associated with squared error loss and cost per unit observation is as small as possible. The design is shown to be asymptotically optimal.
In this paper, we develop an exact method for the determination of the minimum sample size for estimating the proportion of a finite population with prescribed margin of error and confidence level. By characterizing the behavior of the coverage probability with respect to the proportion, we show that the computational complexity can be significantly reduced and bounded regardless population size.
In this paper, a new randomized response technique aimed at protecting respondents' privacy is proposed. It is designed for estimating the population total, or the population mean, of a quantitative characteristic. It provides a~high degree of protection to the interviewed individuals, hence it may be favorably perceived by them and increase their willingness to cooperate. Instead of revealing the true value of the characteristic under investigation, a respondent only states whether the value is greater (or smaller) than a~number which is selected by him/her at random, and is unknown to the in…
Methods for extending -- generalizing or transporting -- inferences from a randomized trial to a target population involve conditioning on a large set of covariates that is sufficient for rendering the randomized and non-randomized groups exchangeable. Yet, decision-makers are often interested in examining treatment effects in subgroups of the target population defined in terms of only a few discrete covariates. Here, we propose methods for estimating subgroup-specific potential outcome means and average treatment effects in generalizability and transportability analyses, using outcome model-b…
Randomized control trials (RCTs) are the gold standard for estimating causal effects, but often use samples that are non-representative of the actual population of interest. We propose a reweighting method for estimating population average treatment effects in settings with noncompliance. Simulations show the proposed compliance-adjusted population estimator outperforms its unadjusted counterpart when compliance is relatively low and can be predicted by observed covariates. We apply the method to evaluate the effect of Medicaid coverage on health care use for a target population of adults who …
Efficient estimation of population size from dependent dual-record system (DRS) remains a statistical challenge in capture-recapture type experiment. Owing to the nonidentifiability of the suitable Time-Behavioral Response Variation model (denoted as $M_{tb}$) under DRS, few methods are developed in Bayesian paradigm based on informative priors. Our contribution in this article is in developing integrated likelihood function from model $M_{tb}$ based on a novel approach developed by Severini (2007, Biometrika). Suitable weight function on nuisance parameter is derived under the assumption of a…
The aim of this paper is to propose a novel estimation method of using genetic-predicted observations to estimate trans-ancestry genetic correlations, which describes how genetic architecture of complex traits varies among populations, in genome-wide association studies (GWAS). Our new estimator corrects for prediction errors caused by high-dimensional weak GWAS signals, while addressing the heterogeneity of GWAS data across ethnicities, such as linkage disequilibrium (LD) differences, which can lead to biased findings in homogeneity-agnostic analyses. Moreover, our estimator only requires one…
Most causal inference studies rely on the assumption of overlap to estimate population or sample average causal effects. When data exhibit non-overlap, estimation of these estimands requires reliance on model specifications, due to poor data support. All existing methods to address non-overlap, such as trimming or down-weighting data in regions of poor support, change the estimand. In environmental health research, where study results are often intended to influence policy, changes in the estimand can diminish the study's impact, because estimates may not be representative of effects in the po…
Powerful array-based single-nucleotide polymorphism--typing platforms have recently heralded a new era in which genome-wide studies are conducted with increasing frequency. A genetic polymorphism associated with population pharmacokinetics (PK) is typically analyzed using nonlinear mixed-effect models (NLMM). Applying NLMM to large-scale data, such as those generated by genome-wide studies, raises several issues related to the assumption of random effects, as follows: (i) Computation time: it takes a long time to compute the marginal likelihood. (ii) Convergence of iterative calculation: an ad…
Dynamic models have been successfully used in producing estimates of HIV epidemics at national level, due to their epidemiological nature and their ability to simultaneously estimate prevalence, incidence, and mortality rates. Recently, HIV interventions and policies have required more information at sub-national and sub-population levels to support local planning, decision making and resource allocation. Unfortunately, many areas and high-risk groups lack sufficient data for deriving stable and reliable results, and this is a critical technical barrier to more stratified estimates. One soluti…
Magnetoencephalography (MEG) is an advanced imaging technique used to measure the magnetic fields outside the human head produced by the electrical activity inside the brain. Various source localization methods in MEG require the knowledge of the underlying active sources, which are identified by a priori. Common methods used to estimate the number of sources include principal component analysis or information criterion methods, both of which make use of the eigenvalue distribution of the data, thus avoiding solving the time-consuming inverse problem. Unfortunately, all these methods are very …
Respondent-Driven Sampling (RDS) is an approach to sampling design and inference in hard-to-reach human populations. Typically, a sampling frame is not available, and population members are difficult to identify or recruit from broader sampling frames. Common examples include injecting drug users, men who have sex with men, and female sex workers. Most analysis of RDS data has focused on estimating aggregate characteristics, such as disease prevalence. However, RDS is often conducted in settings where the population size is unknown and of great independent interest. This paper presents an appr…
Coalescent theory combined with statistical modeling allows us to estimate effective population size fluctuations from molecular sequences of individuals sampled from a population of interest. When sequences are sampled serially through time and the distribution of the sampling times depends on the effective population size, explicit statistical modeling of sampling times improves population size estimation. Previous work assumed that the genealogy relating sampled sequences is known and modeled sampling times as an inhomogeneous Poisson process with log-intensity equal to a linear function of…
We consider an empirical likelihood framework for inference for a statistical model based on an informative sampling design and population-level information. The population-level information is summarized in the form of estimating equations and incorporated into the inference through additional constraints. Covariate information is incorporated both through the weights and the estimating equations. The estimator is based on conditional weights. We show that under usual conditions, with population size increasing unbounded, the estimates are strongly consistent, asymptotically unbiased, and nor…
Challenges of assessing complexity and clonality in populations of mixed species arise in diverse areas of modern biology, including estimating diversity and clonality in microbiome populations, measuring patterns of T and B cell clonality, and determining the underlying tumor cell population structure in cancer. Here we address the problem of quantifying populations, with our analysis directed toward systems for which previously defined algorithms allow the sequence-based identification of clonal subpopulations. Data come from replicate sequencing libraries generated from a sample, potentiall…
Consider the random Dirichlet partition of the interval into $n$ fragments with parameter $θ>0$. We recall the unordered Ewens sampling formulae from finite Dirichlet partitions. As this is a key variable for estimation purposes, focus is on the number of distinct visited species in the sampling process. These are illustrated in specific cases. We use these preliminary statistical results on frequencies distribution to address the following sampling problem: what is the estimated number of species when sampling is from Dirichlet populations? The obtained results are in accordance with the ones…
Non-probability samples become increasingly popular in survey statistics but may suffer from selection biases that limit the generalizability of results to the target population. We consider integrating a non-probability sample with a probability sample which provides high-dimensional representative covariate information of the target population. We propose a two-step approach for variable selection and finite population inference. In the first step, we use penalized estimating equations with folded-concave penalties to select important variables for the sampling score of selection into the no…
This article proposes a novel adaptive design algorithm that can be used to find optimal treatment allocations in N-of-1 clinical trials. This new methodology uses two Laplace approximations to provide a computationally efficient estimate of population and individual random effects within a repeated measures, adaptive design framework. Given the efficiency of this approach, it is also adopted for treatment selection to target the collection of data for the precise estimation of treatment effects. To evaluate this approach, we consider both a simulated and motivating N-of-1 clinical trial from …
Changes in population size influence genetic diversity of the population and, as a result, leave a signature of these changes in individual genomes in the population. We are interested in the inverse problem of reconstructing past population dynamics from genomic data. We start with a standard framework based on the coalescent, a stochastic process that generates genealogies connecting randomly sampled individuals from the population of interest. These genealogies serve as a glue between the population demographic history and genomic sequences. It turns out that only the times of genealogical …
