This paper develops new methodology, together with related theories, for combining information from independent studies through confidence distributions. A formal definition of a confidence distribution and its asymptotic counterpart (i.e., asymptotic confidence distribution) are given and illustrated in the context of combining information. Two general combination methods are developed: the first along the lines of combining p-values, with some notable differences in regard to optimality of Bahadur type efficiency; the second by multiplying and normalizing confidence densities. The latter approach is inspired by the common approach of multiplying likelihood functions for combining parametric information. The paper also develops adaptive combining methods, with supporting asymptotic theory which should be of practical interest. The key point of the adaptive development is that the methods attempt to combine only the correct information, downweighting or excluding studies containing little or wrong information about the true parameter of interest. The combination methodologies are illustrated in simulated and real data examples with a variety of applications.
We consider independent random variables (r.v.'s) with a common mean $μ$ that either satisfy Lindeberg's condition, or are symmetric around $μ$. Present forms of existing functional central limit theorems (FCLT's) for Studentized partial sums of such r.v.'s on $D[0,1]$ are seen to be of some use for constructing asymptotic confidence intervals, or what we call functional asymptotic confidence intervals (FACI's), for $μ$. In this paper we establish completely data-based versions of these FCLT's and thus extend their applicability in this regard. Two special examples of new FACI's for $μ$ are pr…
One-sided confidence intervals are presented for the average of non-identical Bernoulli parameters. These confidence intervals are expressed as analytical functions of the total number of Bernoulli games won, the number of rounds and the confidence level. Tightness of these bounds in the sense of Buehler, i.e. as the strictest possible monotonic intervals, is demonstrated for all confidence levels. A simple interval valid for all confidence levels is also provided with a tightness guarantee. Finally, an application of the proposed confidence intervals to sequential sampling is discussed.
Let $X_1, \ldots, X_n$ be some i.i.d. observations from a heavy tailed distribution $F$, i.e. such that the common distribution of the excesses over a high threshold $u_n$ can be approximated by a Generalized Pareto Distribution $G_{γ,σ_n}$ with $γ>0$. This work is devoted to the problem of finding confidence regions for the couple $(γ,σ_n)$ : combining the empirical likelihood methodology with estimation equations (close but not identical to the likelihood equations) introduced by J. Zhang (Australian and New Zealand J. Stat n.49(1), 2007), asymptotically valid confidence regions for $(γ,σ_n)…
We provide a simple method to combine stochastic bandit algorithms. Our approach is based on a "meta-UCB" procedure that treats each of $N$ individual bandit algorithms as arms in a higher-level $N$-armed bandit problem that we solve with a variant of the classic UCB algorithm. Our final regret depends only on the regret of the base algorithm with the best regret in hindsight. This approach provides an easy and intuitive alternative strategy to the CORRAL algorithm for adversarial bandits, without requiring the stability conditions imposed by CORRAL on the base algorithms. Our results match lo…
Consider a linear regression model with independent and identically normally distributed random errors. Suppose that the parameter of interest is a specified linear combination of the regression parameters. We prove that the usual confidence interval for this parameter is admissible within a broad class of confidence intervals.
Suppose that X_1,X_2,...,X_n are independent and identically Bernoulli(theta) distributed. Also suppose that our aim is to find an exact confidence interval for theta that is the intersection of a 1-α/2 upper confidence interval and a 1-α/2 lower confidence interval. The Clopper-Pearson interval is the standard such confidence interval for theta, which is widely used in practice. We consider the randomized confidence interval of Stevens, 1950 and present some extensions, including pseudorandomized confidence intervals. We also consider the "data-randomized" confidence interval of Korn, 1987 an…
Consider X_1,X_2,...,X_n that are independent and identically N(mu,sigma^2) distributed. Suppose that we have uncertain prior information that mu = 0. We answer the question: to what extent can a frequentist 1-alpha confidence interval for mu utilize this prior information?
I propose a new type of confidence interval for correct asymptotic inference after using data to select a model of interest without assuming any model is correctly specified. This hybrid confidence interval is constructed by combining techniques from the selective inference and post-selection inference literatures to yield a short confidence interval across a wide range of data realizations. I show that hybrid confidence intervals have correct asymptotic coverage, uniformly over a large class of probability distributions that do not bound scaled model parameters. I illustrate the use of these …
In this article, weak convergence of the general non-Markov state transition probability estimator by Titman (2015) is established which, up to now, has not been verified yet for other general non-Markov estimators. A similar theorem is shown for the bootstrap, yielding resampling-based inference methods for statistical functionals. Formulas of the involved covariance functions are presented in detail. Particular applications include the conditional expected length of stay in a specific state, given occupation of another state in the past, as well as the construction of time-simultaneous confi…
Consider a linear regression model with n-dimensional response vector, p-dimensional regression parameter beta and independent normally distributed errors. Suppose that the parameter of interest is theta = a^T beta where a is a specified vector. Define the s-dimensional parameter vector tau=C^T beta-t where C and t are specified. Also suppose that we have uncertain prior information that tau=0. Part of our evaluation of a frequentist confidence interval for theta is the ratio (expected length of this confidence interval)/(expected length of standard 1-alpha confidence interval), the scaled exp…
For any class of one-sided $1-α$ confidence intervals with a certain monotonicity ordering on the random confidence limit, the smallest interval, in the sense of the set inclusion for the difference of two proportions of two independent binomial random variables, is constructed based on a direct analysis of coverage probability function. A special ordering on the confidence limit is developed and the corresponding smallest confidence interval is derived. This interval is then applied to identify the minimum effective dose (MED) for binary data in dose-response studies, and a multiple test proc…
We derive asymptotic theory for the plug-in estimate for density level sets under Hausdoff loss. Based on the asymptotic theory, we propose two bootstrap confidence regions for level sets. The confidence regions can be used to perform tests for anomaly detection and clustering. We also introduce a technique to visualize high dimensional density level sets by combining mode clustering and multidimensional scaling.
Given a sample from some unknown continuous density $f:\mathbb{R}\to\mathbb{R}$, we construct adaptive confidence bands that are honest for all densities in a "generic" subset of the union of $t$-Hölder balls, $0<t\le r$, where $r$ is a fixed but arbitrary integer. The exceptional ("nongeneric") set of densities for which our results do not hold is shown to be nowhere dense in the relevant Hölder-norm topologies. In the course of the proofs we also obtain limit theorems for maxima of linear wavelet and kernel density estimators, which are of independent interest.
Quasi-Monte Carlo confidence intervals using quantiles of randomized nets
Johann Radon Institute for Computational and Applied Mathematics, ÖAW（约翰·拉多纳计算与应用数学研究所，奥地利科学院）；Institute of Fundamental and Transdisciplinary Research, Zhejiang University（浙江大学基础与交叉研究学院）
We demonstrate and discuss nonasymptotic bounds in probability for the cost of a regression scheme with a general loss function from the perspective of the Rademacher theory, and for the optimality with respect to the average $L^{2}$-distance to the underlying conditional expectations of least squares regression outcomes from the perspective of the Vapnik-Chervonenkis theory. The results follow from an analysis involving independent but possibly nonstationary training samples and can be extended, in a manner that we explain and illustrate, to relevant cases in which the training sample exhibit…
To quantify uncertainty around point estimates of conditional objects such as conditional means or variances, parameter uncertainty has to be taken into account. Attempts to incorporate parameter uncertainty are typically based on the unrealistic assumption of observing two independent processes, where one is used for parameter estimation, and the other for conditioning upon. Such unrealistic foundation raises the question whether these intervals are theoretically justified in a realistic setting. This paper presents an asymptotic justification for this type of intervals that does not require …
Computer-Aided Diagnosis has shown stellar performance in providing accurate medical diagnoses across multiple testing modalities (medical images, electrophysiological signals, etc.). While this field has typically focused on fully harvesting the signal provided by a single (and generally extremely reliable) modality, fewer efforts have utilized imprecise data lacking reliable ground truth labels. In this unsupervised, noisy setting, the robustification and quantification of the diagnosis uncertainty become paramount, thus posing a new challenge: how can we combine multiple sources of informat…
This paper is concerned with estimation and inference for the location of a change point in the mean of independent high-dimensional data. Our change point location estimator maximizes a new U-statistic based objective function, and its convergence rate and asymptotic distribution after suitable centering and normalization are obtained under mild assumptions. Our estimator turns out to have better efficiency as compared to the least squares based counterpart in the literature. Based on the asymptotic theory, we construct a confidence interval by plugging in consistent estimates of several quan…
We consider the problem of constructing confidence intervals for the median of a response $Y \in \mathbb{R}$ conditional on features $X \in \mathbb{R}^d$ in a situation where we are not willing to make any assumption whatsoever on the underlying distribution of the data $(X,Y)$. We propose a method based upon ideas from conformal prediction and establish a theoretical guarantee of coverage while also going over particular distributions where its performance is sharp. Additionally, we prove an equivalence between confidence intervals for the conditional median and confidence intervals for the r…
We consider a linear regression model with regression parameter beta =(beta_1, ..., beta_p) and independent and identically N(0, sigma^2)distributed errors. Suppose that the parameter of interest is theta = a^T beta where a is a specified vector. Define the parameter tau = c^T beta - t where the vector c and the number t are specified and a and c are linearly independent. Also suppose that we have uncertain prior information that tau = 0. Kabaila and Giri (2009c) present a new frequentist 1-alpha confidence interval for theta that utilizes this prior information. This interval has expected len…
Practical or scientific considerations often lead to selecting a subset of parameters as ``important.'' Inferences about those parameters often are based on the same data used to select them in the first place. That can make the reported uncertainties deceptively optimistic: confidence intervals that ignore selection generally have less than their nominal coverage probability. Controlling the probability that one or more intervals for selected parameters do not cover---the ``simultaneous over the selected'' (SoS) error rate---is crucial in many scientific problems. Intervals that control the S…
Linear combinations of multinomial probabilities, such as those resulting from contingency tables, are of use when evaluating classification system performance. While large sample inference methods for these combinations exist, small sample methods exist only for regions on the multinomial parameter space instead of the linear combinations. However, in medical classification problems it is common to have small samples necessitating a small sample confidence interval on linear combinations of multinomial probabilities. Therefore, in this paper we derive an exact confidence interval, through the…
In an interesting recent work, Kuzborskij and Szepesvári derived a confidence bound for functions of independent random variables, which is based on an inequality that relates concentration to squared perturbations of the chosen function. Kuzborskij and Szepesvári also established the PAC-Bayes-ification of their confidence bound. Two important aspects of their work are that the random variables could be of unbounded range, and not necessarily of an identical distribution. The purpose of this note is to advertise/discuss these interesting results, with streamlined proofs. This expository note …
This note considers the problem of conducting statistical inference on the share of individuals in some subgroup of a population that experience some event. The specific complication is that the size of the subgroup needs to be estimated, whereas the number of individuals that experience the event is known. The problem is motivated by the recent study of Streeck et al. (2020), who estimate the infection fatality rate (IFR) of SARS-CoV-2 infection in a German town that experienced a super-spreading event in mid-February 2020. In their case the subgroup of interest is comprised of all infected i…
The multivariate Fay-Herriot model is quite effective in combining information through correlations among small area survey estimates of related variables or historical survey estimates of the same variable or both. Though the literature on small area estimation is already very rich, construction of second-order efficient confidence intervals from multivariate models have so far received very little attention. In this paper, we develop a parametric bootstrap method for constructing a second-order efficient confidence interval for a general linear combination of small area means using the multi…
Upper Confidence Bound (UCB) method is arguably the most celebrated one used in online decision making with partial information feedback. Existing techniques for constructing confidence bounds are typically built upon various concentration inequalities, which thus lead to over-exploration. In this paper, we propose a non-parametric and data-dependent UCB algorithm based on the multiplier bootstrap. To improve its finite sample performance, we further incorporate second-order correction into the above construction. In theory, we derive both problem-dependent and problem-independent regret bound…
Confidence intervals for maximum unseen probabilities, with application to sequential sampling design
Bocconi Institute for Data Science and Analytics（博科尼数据科学与分析研究所）；Bocconi University（博科尼大学）；Department of Economics and Statistics, University of Torino（都灵大学经济与统计系）；Collegio Carlo Alberto（卡罗·阿尔贝托学院）
Nonparametric statistics for distribution functions F or densities f=F' under qualitative shape constraints provides an interesting alternative to classical parametric or entirely nonparametric approaches. We contribute to this area by considering a new shape constraint: F is said to be bi-log-concave, if both log(F) and log(1 - F) are concave. Many commonly considered distributions are compatible with this constraint. For instance, any c.d.f. F with log-concave density f = F' is bi-log-concave. But in contrast to the latter constraint, bi-log-concavity allows for multimodal densities. We prov…
We consider a linear regression model with regression parameter beta=(beta_1,...,beta_p) and independent and identically N(0,sigma^2) distributed errors. Suppose that the parameter of interest is theta = a^T beta where a is a specified vector. Define the parameter tau=c^T beta-t where the vector c and the number t are specified and a and c are linearly independent. Also suppose that we have uncertain prior information that tau = 0. We present a new frequentist 1-alpha confidence interval for theta that utilizes this prior information. We require this confidence interval to (a) have endpoints t…
We propose Horvitz-Thompson-like and Hajek-like estimators of the total and mean of the values of a variable of interest associated with the elements of a hard-to-reach population sampled by the variant of link-tracing sampling proposed by Felix-Medina and Thompson (2004). As examples of this type of population are drug users, homeless people and sex workers. In this sampling variant, a frame of venues or places where the members of the population tend to gather, such as parks and bars, is constructed. The frame is not assumed to cover the whole population. An initial cluster sample of element…
Given $m$ unknown parameters with corresponding independent estimators, the Benjamini-Hochberg (BH) procedure can be used to classify the sign of parameters such that the expected proportion of erroneous directional decisions (directional FDR) is controlled at a preset level $q$. More ambitiously, our goal is to construct sign-determining confidence intervals---instead of only classifying the sign---such that the expected proportion of non-covering constructed intervals (FCR) is controlled. We suggest a valid procedure which adjusts a marginal confidence interval in order to construct a maximu…
In this article, we derive and compare methods to derive \textit{p}-values and sets of confidence intervals with strong control of the family-wise error rates and coverage for estimates of treatment effects in cluster randomised trials with multiple outcomes. There are few methods for \textit{p}-value corrections and deriving confidence intervals, limiting their application in this setting. We discuss the methods of Bonferroni, Holm, and Romano \& Wolf (2005) and adapt them to cluster randomised trial inference using permutation-based methods with different test statistics. We develop a novel …
Survey respondents may give untruthful answers to sensitive questions when asked directly. In recent years, researchers have turned to the list experiment (also known as the item count technique) to overcome this difficulty. While list experiments may be less prone to bias than direct questioning, list experiments are also more susceptible to sampling variability. We show that researchers do not have to abandon direct questioning altogether in order to gain the advantages of list experimentation. We develop a nonparametric estimator of the prevalence of sensitive behaviors that combines list e…
A standard approach to confidence intervals for quantiles requires good estimates of the quantile density. The optimal bandwidth for kernel estimation of the quantile density depends on an underlying location-scale family only through the quantile optimality ratio (QOR), which is the starting point for our results. While the QOR is not distribution-free, it turns out that what is optimal for one family often works quite well for families having similar shape. This allows one to rely on a single representative QOR if one has a rough idea of the distributional shape. Another option that we explo…
Deep learning architectures have proved versatile in a number of drug discovery applications, including the modelling of in vitro compound activity. While controlling for prediction confidence is essential to increase the trust, interpretability and usefulness of virtual screening models in drug discovery, techniques to estimate the reliability of the predictions generated with deep learning networks remain largely underexplored. Here, we present Deep Confidence, a framework to compute valid and efficient confidence intervals for individual predictions using the deep learning technique Snapsho…
Good's formula and Fisher's method are frequently used for combining independent P-values. Interestingly, the equivalent of Good's formula already emerged in 1910 and mathematical expressions relevant to even more general situations have been repeatedly derived, albeit in different context. We provide here a novel derivation and show how the analytic formula obtained reduces to the two aforementioned ones as special cases. The main novelty of this paper, however, is the explicit treatment of nearly degenerate weights, which are known to cause numerical instabilities. We derive a controlled exp…
We propose confidence regions with asymptotically correct uniform coverage probability of parameters whose Fisher information matrix can be singular at important points of the parameter set. Our work is motivated by the need for reliable inference on scale parameters close or equal to zero in mixed models, which is obtained as a special case. The confidence regions are constructed by inverting a continuous extension of the score test statistic standardized by expected information, which we show exists at points of singular information under regularity conditions. Similar results have previousl…
The flexibility and wide applicability of the Fisher randomization test (FRT) makes it an attractive tool for assessment of causal effects of interventions from modern-day randomized experiments that are increasing in size and complexity. This paper provides a theoretical inferential framework for FRT by establishing its connection with confidence distributions Such a connection leads to development of (i) an unambiguous procedure for inversion of FRTs to generate confidence intervals with guaranteed coverage, (ii) generic and specific methods to combine FRTs from multiple independent experime…
Reacting against the limitation of statistics to decision procedures, R. A. Fisher proposed for inductive reasoning the use of the fiducial distribution, a parameter-space distribution of epistemological probability transferred directly from limiting relative frequencies rather than computed according to the Bayes update rule. The proposal is developed as follows using the confidence measure of a scalar parameter of interest. (With the restriction to one-dimensional parameter space, a confidence measure is essentially a fiducial probability distribution free of complications involving ancillar…
We study the problem of identifying the source of a diffusion spreading over a regular tree. When the degree of each node is at least three, we show that it is possible to construct confidence sets for the diffusion source with size independent of the number of infected nodes. Our estimators are motivated by analogous results in the literature concerning identification of the root node in preferential attachment and uniform attachment trees. At the core of our proofs is a probabilistic analysis of Pólya urns corresponding to the number of uninfected neighbors in specific subtrees of the infect…
