Optimal sequential multiple hypothesis tests
Andrey Novikov
本文讨论了离散时间随机过程分布的多假设检验问题。贝叶斯和条件设置被认为是。刻画了最优序贯检验的结构。
This work deals with a general problem of testing multiple hypotheses about the distribution of a discrete-time stochastic process. Both the Bayesian and the conditional settings are considered. The structure of optimal sequential tests is characterized.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We present a test for the problem of decentralized sequential hypothesis testing, which is asymptotically optimum. By selecting a suitable sampling mechanism at each sensor, communication between sensors and fusion center is asynchronous and limited to 1-bit data. The proposed SPRT-like test turns out to be order-2 asymptotically optimum in the case of continuous time and continuous path signals, while in discrete time this strong asymptotic optimality property is preserved under proper conditions. If these conditions do not hold, then we can show optimality of order-1. Simulations corroborate…
Sequential multiple hypothesis testing in presence of control variables
假设在统计实验的任何阶段，可以使用控制变量$X$，该控制变量影响在该阶段观察到的数据$Y$的分布。$Y$的分布取决于一些未知的参数$θ$，我们考虑的问题，测试多个假设$H_1: θ=θ_1$，$H_2: θ=θ_2, ...$，$H_k: θ=θ_k$允许数据被控制的$X$，在以下顺序的上下文中。实验开始时，将值$X_1$分配给控制变量，并观察$Y_1$作为响应。在一些分析之后，选择控制变量的另一个值$X_2$，并且观察作为响应的$Y_2$，等等。假设实验最终停止，并且在那一刻，将采取有利于假设之一的最终决定$H_1,...$，$H_k$。在这篇文章中，我们的目的是表征结构的最佳序贯测试程序的基础上，从这种类型的实验中获得的数据的情况下，当观察$Y_1, Y_2,..., Y_n$是独立的，控制$X_1,X_2,..., X_n$，$n=1,2,...$。
The sequential hypothesis testing problem is a class of statistical analyses where the sample size is not fixed in advance. Instead, the decision-process takes in new observations sequentially to make real-time decisions for testing an alternative hypothesis against a null hypothesis until some stopping criterion is satisfied. In many common applications of sequential hypothesis testing, the data can be highly sensitive and may require privacy protection; for example, sequential hypothesis testing is used in clinical trials, where doctors sequentially collect data from patients and must determ…
Under some mild Markov assumptions it is shown that the problem of designing optimal sequential tests for two simple hypotheses can be formulated as a linear program. The result is derived by investigating the Lagrangian dual of the sequential testing problem, which is an unconstrained optimal stopping problem, depending on two unknown Lagrangian multipliers. It is shown that the derivative of the optimal cost function with respect to these multipliers coincides with the error probabilities of the corresponding sequential test. This property is used to formulate an optimization problem that is…
Under mild Markov assumptions, sufficient conditions for strict minimax optimality of sequential tests for multiple hypotheses under distributional uncertainty are derived. First, the design of optimal sequential tests for simple hypotheses is revisited and it is shown that the partial derivatives of the corresponding cost function are closely related to the performance metrics of the underlying sequential test. Second, an implicit characterization of the least favorable distributions for a given testing policy is stated. By combining the results on optimal sequential tests and least favorable…
The $γ$-FDP and $k$-FWER multiple testing error metrics, which are tail probabilities of the respective error statistics, have become popular recently as less-stringent alternatives to the FDR and FWER. We propose general and flexible stepup and stepdown procedures for testing multiple hypotheses about sequential (or streaming) data that simultaneously control both the type I and II versions of $γ$-FDP, or $k$-FWER. The error control holds regardless of the dependence between data streams, which may be of arbitrary size and shape. All that is needed is a test statistic for each data stream tha…
Consider a decision maker who is responsible to collect observations so as to enhance his information in a speedy manner about an underlying phenomena of interest. The policies under which the decision maker selects sensing actions can be categorized based on the following two factors: i) sequential vs. non-sequential; ii) adaptive vs. non-adaptive. Non-sequential policies collect a fixed number of observation samples and make the final decision afterwards; while under sequential policies, the sample size is not known initially and is determined by the observation outcomes. Under adaptive poli…
In this paper, we propose a computer-oriented method of construction of optimal group sequential hypothesis tests with variable group sizes. In particular, for independent and identically distributed observations we obtain the form of optimal group sequential tests which turn to be a particular case of sequentially planned probability ratio tests (SPPRTs, Schmitz, 1993) . Formulas are given for computing the numerical characteristics of general SPPRTs, like error probabilities, average sampling cost, etc. A numerical method of designing the optimal tests and evaluation of the performance chara…
Instance-Log-Optimality of Portfolio-Based E-Processes and their Sequential Hypothesis Tests
本文针对基于$e$-过程的序贯假设检验，证明满足次线性遗憾界的$e$-过程在复合备择假设下渐近且几乎必然实例对数最优，并给出匹配的预期拒绝时间上下界，且结果在分布一致意义上成立。
We consider sequential hypothesis testing based on observations which are received in groups of random size. The observations are assumed to be independent both within and between the groups. We assume that the group sizes are independent and their distributions are known, and that the groups are formed independently of the observations. We are concerned with a problem of testing a simple hypothesis against a simple alternative. For any (group-) sequential test, we take into account the following three characteristics: its type I and type II error probabilities and the average cost of observat…
In this paper we present a sequential hypothesis test for the detection of general jump size distrubution. Infinitesimal generators for the corresponding log-likelihood ratios are presented and analyzed. Bounds for infinitesimal generators in terms of super-solutions and sub-solutions are computed. This is shown to be implementable in relation to various classification problems for a crude oil price data set. Machine and deep learning algorithms are implemented to extract a specific deterministic component from the crude oil data set, and the deterministic component is implemented to improve t…
We consider the classical sequential binary hypothesis testing problem in which there are two hypotheses governed respectively by distributions $P_0$ and $P_1$ and we would like to decide which hypothesis is true using a sequential test. It is known from the work of Wald and Wolfowitz that as the expectation of the length of the test grows, the optimal type-I and type-II error exponents approach the relative entropies $D(P_1\|P_0)$ and $D(P_0\|P_1)$. We refine this result by considering the optimal backoff---or second-order asymptotics---from the corner point of the achievable exponent region …
Sequential multiple testing with multiple hypotheses and prior information on the hypothesis configuration
针对多个独立序贯观测数据流中每个流有多个候选假设的问题，利用假设配置的先验信息，设计了一种可靠、计算高效且渐近最优的序贯检验程序。
Optimal Error Exponents for Composite Sequential Quantum Hypothesis Testing
本文研究了复合序列量子假设检验问题，提出了一种混合-序列量子概率比检验方法，通过自适应选择测量并停止阈值跨越，同时达到类型I和最坏情况类型II误差指数。
We consider the problem of simultaneous detection and estimation under a sequential framework. In particular we are interested in sequential tests that distinguish between the null and the alternative hypothesis and every time the decision is in favor of the alternative they provide an estimate of a random parameter. As we demonstrate with our analysis treating the two subproblems separately with the corresponding optimal strategies does not result in the best possible performance. To enjoy optimality one needs to take into account the optimum estimator during the hypothesis testing phase.
Sequential One-Sided Hypothesis Testing of Markov Chains
针对马尔可夫链的序贯检验问题，提出基于数据驱动估计器的单侧SPRT型检验，在估计器遗憾为对数阶时渐近匹配最优性能，无需先验信息。
In this article, we study the hypothesis testing of the blip / net effects of treatments in a treatment sequence. We illustrate that the likelihood ratio test and the score test may suffer from the curse of dimensionality, the null paradox and the high-dimensional constraint on standard parameters under the null hypothesis. On the other hand, we construct the Wald test via a small number of point effects of treatments in single-point causal inference. We show that the Wald test can avoid these problems under the same assumptions as the Wald test for testing the point effect of treatment. The s…
We consider sequential hypothesis testing between two quantum states using adaptive and non-adaptive strategies. In this setting, samples of an unknown state are requested sequentially and a decision to either continue or to accept one of the two hypotheses is made after each test. Under the constraint that the number of samples is bounded, either in expectation or with high probability, we exhibit adaptive strategies that minimize both types of misidentification errors. Namely, we show that these errors decrease exponentially (in the stopping time) with decay rates given by the measured relat…
In this article, a general problem of sequential statistical inference for general discrete-time stochastic processes is considered. The problem is to minimize an average sample number given that Bayesian risk due to incorrect decision does not exceed some given bound. We characterize the form of optimal sequential stopping rules in this problem. In particular, we have a characterization of the form of optimal sequential decision procedures when the Bayesian risk includes both the loss due to incorrect decision and the cost of observations.
In the problem of active sequential hypothesis testing (ASHT), a learner seeks to identify the true hypothesis from among a known set of hypotheses. The learner is given a set of actions and knows the random distribution of the outcome of any action under any true hypothesis. Given a target error $δ>0$, the goal is to sequentially select the fewest number of actions so as to identify the true hypothesis with probability at least $1 - δ$. Motivated by applications in which the number of hypotheses or actions is massive (e.g., genomics-based cancer detection), we propose efficient (greedy, in fa…
Granger Causality Detection via Sequential Hypothesis Testing
针对测量缺陷影响因果检测性能的问题，提出基于向量自回归系数范数的序贯假设检验方法，用于序贯检测两时间序列间的Granger因果关系，并通过仿真验证了其有效性。
We consider the sequential composite binary hypothesis testing problem in which one of the hypotheses is governed by a single distribution while the other is governed by a family of distributions whose parameters belong to a known set $Γ$. We would like to design a test to decide which hypothesis is in effect. Under the constraints that the probabilities that the length of the test, a stopping time, exceeds $n$ are bounded by a certain threshold $ε$, we obtain certain fundamental limits on the asymptotic behavior of the sequential test as $n$ tends to infinity. Assuming that $Γ$ is a convex an…
Universal outlier hypothesis testing is studied in a sequential setting. Multiple observation sequences are collected, a small subset of which are outliers. A sequence is considered an outlier if the observations in that sequence are generated by an "outlier" distribution, distinct from a common "typical" distribution governing the majority of the sequences. Apart from being distinct, the outlier and typical distributions can be arbitrarily close. The goal is to design a universal test to best discern all the outlier sequences. A universal test with the flavor of the repeated significance test…
This paper addresses the following general scenario: A scientist wishes to perform a battery of experiments, each generating a sequential stream of data, to investigate some phenomenon. The scientist would like to control the overall error rate in order to draw statistically-valid conclusions from each experiment, while being as efficient as possible. The between-stream data may differ in distribution and dimension but also may be highly correlated, even duplicated exactly in some cases. Treating each experiment as a hypothesis test and adopting the familywise error rate (FWER) metric, we give…
Adjusted inference for multiple testing procedure in group sequential designs
针对成组序贯设计中同时调整多重假设检验与多次分析的研究空白，提出调整序贯p值及交集假设序贯p值，通过加权Bonferroni和加权参数检验控制族系I类错误率进行推断。
Multi-Armed Sequential Hypothesis Testing by Betting
本文研究了通过赌注进行多臂顺序检验的问题，提出了一种在多个数据源（臂）中选择以获取数据的统计学家的变体，旨在拒绝全局空假设P（所有臂在某种意义上无效）并支持复合替代假设Q（至少有一个臂非空）。通过推广对数最优性和期望拒绝时间最优性的概念，得到了匹配的上下界，并提出了一个修改的上置信界算法来处理不可观测但足够可估计的奖励。
Quantum Sequential Universal Hypothesis Testing
针对复合量子假设检验中固定副本协议效率不足的问题，提出融合自适应局部测量与优化联合测量的 QSUT 框架，可严格控制 I 类错误并降低副本复杂度。
Conventional multiple hypothesis tests use step-up, step-down, or closed testing methods to control the overall error rates. We will discuss marrying these methods with adaptive multistage sampling rules and stopping rules to perform efficient multiple hypothesis testing in sequential experimental designs. The result is a multistage step-down procedure that adaptively tests multiple hypotheses while preserving the family-wise error rate, and extends Holm's (1979) step-down procedure to the sequential setting, yielding substantial savings in sample size with small loss in power.
Policy Design for Active Sequential Hypothesis Testing using Deep Learning
本文研究了如何利用深度学习设计更有效的主动顺序假设检验策略，通过比较新提出的启发式方法与现有方法，展示了在某些场景下性能的显著提升。
Hypothesis testing is an important problem with applications in target localization, clinical trials etc. Many active hypothesis testing strategies operate in two phases: an exploration phase and a verification phase. In the exploration phase, selection of experiments is such that a moderate level of confidence on the true hypothesis is achieved. Subsequent experiment design aims at improving the confidence level on this hypothesis to the desired level. In this paper, the focus is on the verification phase. A confidence measure is defined and active hypothesis testing is formulated as a confid…
This work investigates the sequential hypothesis testing problem with online sensor selection and sensor usage constraints. That is, in a sensor network, the fusion center sequentially acquires samples by selecting one "most informative" sensor at each time until a reliable decision can be made. In particular, the sensor selection is carried out in the online fashion since it depends on all the previous samples at each time. Our goal is to develop the sequential test (i.e., stopping rule and decision function) and sensor selection strategy that minimize the expected sample size subject to the …
A/B tests are typically analyzed via frequentist p-values and confidence intervals; but these inferences are wholly unreliable if users endogenously choose samples sizes by *continuously monitoring* their tests. We define *always valid* p-values and confidence intervals that let users try to take advantage of data as fast as it becomes available, providing valid statistical inference whenever they make their decision. Always valid inference can be interpreted as a natural interface for a sequential hypothesis test, which empowers users to implement a modified test tailored to them. In particul…
A Numerical Approach to Sequential Multi-Hypothesis Testing for Bernoulli Model
本文针对伯努利模型提出基于拉格朗日乘子法的序贯多假设检验数值方法，通过最小化加权期望样本量并在R中实现算法，在三假设情形下验证了其最优性并与矩阵序贯概率比检验进行了比较。
A class of sequential multi-hypothesis tests
本文研究多重假设的序贯检验，基于后向归纳法提出一种简化的 DBC 检验方法，在多个数值实验中表现出相对于最优 Bayesian 检验和 MSPRT 的高效率。
Robust Sequential Hypothesis Testing with Generalized Estimating Equations for Incomplete Clustered and Longitudinal Data
本文提出了一种基于广义估计方程的稳健序列假设检验方法，适用于不完整聚类和纵向数据，通过不依赖建模假设来提高鲁棒性，并改进功效边界计算与多重插补应用。
We discuss an "operational" approach to testing convex composite hypotheses when the underlying distributions are heavy-tailed. It relies upon Euclidean separation of convex sets and can be seen as an extension of the approach to testing by convex optimization developed in [8, 12]. In particular, we show how one can construct quasi-optimal testing procedures for families of distributions which are majorated, in a certain precise sense, by a sub-spherical symmetric one and study the relationship between tests based on Euclidean separation and "potential-based tests." We apply the promoted metho…
The problem of joint sequential detection and isolation is considered in the context of multiple, not necessarily independent, data streams. A multiple testing framework is proposed, where each hypothesis corresponds to a different subset of data streams, the sample size is a stopping time of the observations, and the probabilities of four kinds of error are controlled below distinct, user-specified levels. Two of these errors reflect the detection component of the formulation, whereas the other two the isolation component. The optimal expected sample size is characterized to a first-order asy…
Beyond boundaries: Gary Lorden's groundbreaking contributions to sequential analysis
本文回顾了 Gary Lorden 在序贯假设检验和变点检测中的基础性贡献及其后续扩展，并介绍了他在 NASA、娱乐业和棒球等领域的重要咨询工作。
In this paper, two tests, based on CUSUM of the residuals and least squares estimation, are studied to detect in real time a change-point in a nonlinear model. A first test statistic is proposed by extension of a method already used in the literature but for the linear models. It is tested the null hypothesis, at each sequential observation, that there is no change in the model against a change presence. The asymptotic distribution of the test statistic under the null hypothesis is given and its convergence in probability to infinity is proved when a change occurs. These results will allow to …
A studentized permutation test in group sequential designs
本文提出一种学生化置换检验，用于分组序贯设计，以改善小样本下的第一类错误率控制，同时保持渐近性质，模拟研究验证了其有效性。
The problem of multi-hypothesis testing with controlled sensing of observations is considered. The distribution of observations collected under each control is assumed to follow a single-parameter exponential family distribution. The goal is to design a policy to find the true hypothesis with minimum expected delay while ensuring that the probability of error is below a given constraint. The decision-maker can control the delay by intelligently choosing the control for observation collection in each time slot. We derive a policy that satisfies the given constraint on the error probability. We …
We propose a new algorithmic framework for sequential hypothesis testing with i.i.d. data, which includes A/B testing, nonparametric two-sample testing, and independence testing as special cases. It is novel in several ways: (a) it takes linear time and constant space to compute on the fly, (b) it has the same power guarantee as a non-sequential version of the test with the same computational constraints up to a small factor, and (c) it accesses only as many samples as are required - its stopping time adapts to the unknown difficulty of the problem. All our test statistics are constructed to b…
This work analyzes the asymptotic performances of fully distributed sequential hypothesis testing procedures as the type-I and type-II error rates approach zero, in the context of a sensor network without a fusion center. In particular, the sensor network is defined by an undirected graph, where each sensor can observe samples over time, access the information from the adjacent sensors, and perform the sequential test based on its own decision statistic. Different from most literature, the sampling process and the information exchange process in our framework take place simultaneously (or, at …
Classifier-Based Nonparametric Sequential Hypothesis Testing
本文提出了一种基于分类器的非参数序列假设检验方法，通过离线数据集训练多类分类器，设计水平α的检验方法，并分析了检验停止时间的上界及分类器分离条件。
Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams
提出基于赌博框架的非参数序贯检验方法PEAK，用于多数据流均值复合假设检验，在保证第一类错误控制的同时显著减少样本量并降低计算复杂度。
E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
提出E-valuator方法，将任意黑盒验证器分数转化为具有可控虚警率的决策规则，通过序贯假设检验实现对智能体轨迹的在线监控，提升统计功效并节省令牌。
We consider the sequential change-point detection problem of detecting changes that are characterized by a subspace structure. Such changes are frequent in high-dimensional streaming data altering the form of the corresponding covariance matrix. In this work we present a Subspace-CUSUM procedure and demonstrate its first-order asymptotic optimality properties for the case where the subspace structure is unknown and needs to be simultaneously estimated. To achieve this goal we develop a suitable analytical methodology that includes a proper parameter optimization for the proposed detection sche…
Precision and Decisiveness as Goals: Reliable Sequential Hypothesis Testing with a Dual Stopping Criterion
该研究提出DPitG方法，结合精度目标与明确裁决，将PitG的不确定率从62%降至2%且零假阳性，符合预注册标准，适用于需可靠裁决的场景。
Human-AI-Powered Hypothesis Testing: Cost-Aware Selective AI Scoring and Sequential Human Escalation
本研究提出SCALE策略，结合选择性AI评分与序贯人工升级，在控制统计检验错误的同时最小化成本，并通过信息论下界验证其最优性。
Optimal Sequential Detection by Sparsity Likelihood
本文研究多数据流序贯变点检测，给出不同稀疏性水平下检测延迟的渐近下界，并提出基于稀疏似然得分的停止规则以实现最优检测，数值实验验证了其有效性。
Recent developments towards optimality in multiple hypothesis testing
即使在测试一个假设时，也有许多不同的最优性概念。在多重测试领域，可能性的数量要大得多。本文首先将描述涉及单个参数的测试中出现的多重性问题，并将在此背景下描述一个新的最优性结果。虽然给出的例子是最小的实际重要性，它说明了至关重要的依赖最优的测试问题的精确规格。然后，本文将讨论类型的扩展的最优性准则，被认为是假设涉及多个参数时，将注意到一些新的最优性结果，并会选择相关的理论参考这些扩展的标准下的最优性考虑。
The goal of the paper is to develop a specific application of the convex optimization based hypothesis testing techniques developed in A. Juditsky, A. Nemirovski, "Hypothesis testing via affine detectors," Electronic Journal of Statistics 10:2204--2242, 2016. Namely, we consider the Change Detection problem as follows: given an evolving in time noisy observations of outputs of a discrete-time linear dynamical system, we intend to decide, in a sequential fashion, on the null hypothesis stating that the input to the system is a nuisance, vs. the alternative stating that the input is a "nontrivia…
We consider the question of sequential prediction under the log-loss in terms of cumulative regret. Namely, given a hypothesis class of distributions, learner sequentially predicts the (distribution of the) next letter in sequence and its performance is compared to the baseline of the best constant predictor from the hypothesis class. The well-specified case corresponds to an additional assumption that the data-generating distribution belongs to the hypothesis class as well. Here we present results in the more general misspecified case. Due to special properties of the log-loss, the same probl…
Consider a decision maker who is responsible to dynamically collect observations so as to enhance his information about an underlying phenomena of interest in a speedy manner while accounting for the penalty of wrong declaration. Due to the sequential nature of the problem, the decision maker relies on his current information state to adaptively select the most ``informative'' sensing action among the available ones. In this paper, using results in dynamic programming, lower bounds for the optimal total cost are established. The lower bounds characterize the fundamental limits on the maximum a…
This paper considers the sequential composite hypothesis test with multiple sensors. The sensors observe random samples in parallel and communicate with a fusion center, who makes the global decision based on the sensor inputs. On one hand, in the centralized scenario, where local samples are precisely transmitted to the fusion center, the generalized sequential likelihood ratio test (GSPRT) is shown to be asymptotically optimal in terms of the expected sample size as error rates tend to zero. On the other hand, for systems with limited power and bandwidth resources, decentralized solutions th…
We consider a unified framework of sequential change-point detection and hypothesis testing modeled by means of hidden Markov chains. One observes a sequence of random variables whose distributions are functionals of a hidden Markov chain. The objective is to detect quickly the event that the hidden Markov chain leaves a certain set of states, and to identify accurately the class of states into which it is absorbed. We propose computationally tractable sequential detection and identification strategies and obtain sufficient conditions for the asymptotic optimality in two Bayesian formulations.…
True and false discoveries with independent and sequential e-values
该研究针对多重假设检验场景，提出基于独立与序贯e值的新算法，经模拟、实证及理论验证，其性能优于现有独立p值算法及无独立性假设的e值算法。
We address the following question in this paper: "What are the most robust statistical methods for social choice?'' By leveraging the theory of uniformly least favorable distributions in the Neyman-Pearson framework to finite models and randomized tests, we characterize uniformly most powerful (UMP) tests, which is a well-accepted statistical optimality w.r.t. robustness, for testing whether a given alternative is the winner under Mallows' model and under Condorcet's model, respectively.
We study a class of distributed hypothesis testing against conditional independence problems. Under the criterion that stipulates minimization of the Type II error rate subject to a (constant) upper bound $ε$ on the Type I error rate, we characterize the set of encoding rates and exponent for both discrete memoryless and memoryless vector Gaussian settings.
Predicting fixed-sample test decisions enables anytime-valid inference
研究针对数据顺序到达时固定样本假设检验的问题，提出将其转变为随时有效检验的程序，通过预测固定样本量下拒绝原假设概率并阈值化得停止规则，能控制第一类错误、节省样本量，对临床试验等有重要意义。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。