We elaborate on a deconvolution method, used to estimate the empirical distribution of unknown parameters, as suggested recently by Efron (2013). It is applied to estimating the empirical distribution of the 'sampling probabilities' of m sampled items. The estimated empirical distribution is used to modify the Horvitz-Thompson estimator. The performance of the modified Horvitz-Thompson estimator is studied in two examples. In one example the sampling probabilities are estimated based on the number of visits until a response was obtained. The other example is based on real data from panel sampling, where in four consecutive months there are corresponding four attempts to interview each member in a panel. The sampling probabilities are estimated based on the number of successful attempts.
We also discuss briefly, further applications of deconvolution, including estimation of False discovery rate.
When there is interference, a subject's outcome depends on the treatment of others and treatment effects may take on several different forms. This situation arises often, particularly in vaccine evaluation. In settings where interference is likely, two-stage cluster randomized trials have been suggested as a means of estimating some of the causal contrast of interest. Working in the finite population setting to investigate rare and unplanned subgroup analyses using some of the estimators that have been suggested in the literature, include Horvitz-Thompson, Hajek, and what might be called the n…
In this paper, we develop a general approach to proving global and local uniform limit theorems for the Horvitz-Thompson empirical process arising from complex sampling designs. Global theorems such as Glivenko-Cantelli and Donsker theorems, and local theorems such as local asymptotic modulus and related ratio-type limit theorems are proved for both the Horvitz-Thompson empirical process, and its calibrated version. Limit theorems of other variants and their conditional versions are also established. Our approach reveals an interesting feature: the problem of deriving uniform limit theorems fo…
When collections of functional data are too large to be exhaustively observed, survey sampling techniques provide an effective way to estimate global quantities such as the population mean function. Assuming functional data are collected from a finite population according to a probabilistic sampling scheme, with the measurements being discrete in time and noisy, we propose to first smooth the sampled trajectories with local polynomials and then estimate the mean function with a Horvitz-Thompson estimator. Under mild conditions on the population size, observation times, regularity of the trajec…
The generalization ability of minimizers of the empirical risk in the context of binary classification has been investigated under a wide variety of complexity assumptions for the collection of classifiers over which optimization is performed. In contrast, the vast majority of the works dedicated to this issue stipulate that the training dataset used to compute the empirical risk functional is composed of i.i.d. observations. Beyond the cases where training data are drawn uniformly without replacement among a large i.i.d. sample or modelled as a realization of a weakly dependent sequence of r.…
For fixed size sampling designs with high entropy it is well known that the variance of the Horvitz-Thompson estimator can be approximated by the Hájek formula. The interest of this asymptotic variance approximation is that it only involves the first order inclusion probabilities of the statistical units. We extend this variance formula when the variable under study is functional and we prove, under general conditions on the regularity of the individual trajectories and the sampling design, that we can get a uniformly convergent estimator of the variance function of the Horvitz-Thompson estima…
This paper addresses the survey estimation of a population mean in continuous time. For this purpose we extend the rotation sampling method to functional data. In contrast to conventional rotation designs that select the sample before the survey, our approach randomizes each sample replacement and thus allows for adaptive sampling. Using Markov chain theory, we evaluate the covariance structure and the integrated squared error [ISE] of the related Horvitz-Thompson estimator. Our sampling designs decrease the mean ISE by suitably reallocating the sample across population strata during replaceme…
The cross-classified sampling design consists in drawing samples from a two-dimension population, independently in each dimension. Such design is commonly used in consumer price index surveys and has been recently applied to draw a sample of babies in the French ELFE survey, by crossing a sample of maternity units and a sample of days. We propose to derive a general theory of estimation for this sampling design. We consider the Horvitz-Thompson estimator for a total, and show that the cross-classified design will usually result in a loss of efficiency as compared to the widespread two-stage de…
We prove that any implementation of pivotal sampling is more efficient than multinomial sampling. This property entails the weak consistency of the Horvitz-Thompson estimator and the existence of a conservative variance estimator. A small simulation study supports our findings.
The paper proposes some robust estimators of the finite population mean. Such estimators are particularly suitable in the presence of some outlying observations. Included as special cases of our general result are robust versions of the ratio estimator and the Horvitz-Thompson estimator. The robust estimators are derived on the basis of certain predictive influence functions.
For a joint model-based and design-based inference, we establish functional central limit theorems for the Horvitz-Thompson empirical process and the Hájek empirical process centered by their finite population mean as well as by their super-population mean in a survey sampling framework. The results apply to single-stage unequal probability sampling designs and essentially only require conditions on higher order correlations. We apply our main results to a Hadamard differentiable statistical functional and illustrate its limit behavior by means of a computer simulation.
Bipartite incidence graph sampling provides a unified representation of many sampling situations for the purpose of estimation, including the existing unconventional sampling methods, such as indirect, network or adaptive cluster sampling, which are not originally described as graph problems. We develop a large class of linear estimators based on the edges in the sample bipartite incidence graph, subjected to a general condition of design unbiasedness. The class contains as special cases the classic Horvitz-Thompson estimator, as well as the other unbiased estimators in the literature of uncon…
We extend the problem of obtaining an estimator for the finite population mean parameter incorporating complete auxiliary information through calibration estimation in survey sampling but considering a functional data framework. The functional calibration sampling weights of the estimator are obtained by matching the calibration estimation problem with the maximum entropy on the mean principle. In particular, the calibration estimation is viewed as an infinite dimensional linear inverse problem following the structure of the maximum entropy on the mean approach. We give a precise theoretical s…
Applied researchers often construct a network from a random sample of nodes in order to infer properties of the parent network. Two of the most widely used sampling schemes are subgraph sampling, where we sample each vertex independently with probability $p$ and observe the subgraph induced by the sampled vertices, and neighborhood sampling, where we additionally observe the edges between the sampled vertices and their neighbors. In this paper, we study the problem of estimating the number of motifs as induced subgraphs under both models from a statistical perspective. We show that: for any co…
Random sampling is an essential tool in the processing and transmission of data. It is used to summarize data too large to store or manipulate and meet resource constraints on bandwidth or battery power. Estimators that are applied to the sample facilitate fast approximate processing of queries posed over the original data and the value of the sample hinges on the quality of these estimators. Our work targets data sets such as request and traffic logs and sensor measurements, where data is repeatedly collected over multiple {\em instances}: time periods, locations, or snapshots. We are interes…
Random samples are lossy summaries which allow queries posed over the data to be approximated by applying an appropriate estimator to the sample. The effectiveness of sampling, however, hinges on estimator selection. The choice of estimators is subjected to global requirements, such as unbiasedness and range restrictions on the estimate value, and ideally, we seek estimators that are both efficient to derive and apply and {\em admissible} (not dominated, in terms of variance, by other estimators). Nevertheless, for a given data domain, sampling scheme, and query, there are many admissible esti…
