Inference on the conditional mean function (CMF) is central to tasks from adaptive experimentation to optimal treatment assignment and algorithmic fairness auditing. In this work, we provide a novel asymptotic anytime-valid test for a CMF global null (e.g., that all conditional means are zero) and contrasts between CMFs, enabling experimenters to make high confidence decisions at any time during the experiment beyond a minimum sample size. We provide mild conditions under which our tests achieve (i) asymptotic type-I error guarantees, (i) power one, and, unlike past tests, (iii) optimal sample complexity relative to a Gaussian location testing. By inverting our tests, we show how to construct function-valued asymptotic confidence sequences for the CMF and contrasts thereof. Experiments on both synthetic and real-world data show our method is well-powered across various distributions while preserving the nominal error rate under continuous monitoring.
We propose a sequential, anytime-valid method to test the conditional independence of a response $Y$ and a predictor $X$ given a random vector $Z$. The proposed test is based on e-statistics and test martingales, which generalize likelihood ratios and allow valid inference at arbitrary stopping times. In accordance with the recently introduced model-X setting, our test depends on the availability of the conditional distribution of $X$ given $Z$, or at least a sufficiently sharp approximation thereof. Within this setting, we derive a general method for constructing e-statistics for testing cond…
Predicting fixed-sample test decisions enables anytime-valid inference
AI & Robotics Institute, Ellison Institute of Technology（艾利森技术学院人工智能与机器人研究所）；Department of Statistics, University of Oxford（牛津大学统计系）；University of Kent（肯特大学）
E-variables are tools for retaining type-I error guarantee with optional stopping. We extend E-variables for sequential two-sample tests to general null hypotheses and anytime-valid confidence sequences. We provide implementations for estimating risk difference, relative risk and odds-ratios in contingency tables.
Confidence sequences, anytime p-values (called p-processes in this paper), and e-processes all enable sequential inference for composite and nonparametric classes of distributions at arbitrary stopping times. Examining the literature, one finds that at the heart of all these (quite different) approaches has been the identification of nonnegative (super)martingales. Thus, informally, nonnegative (super)martingales are known to be sufficient for \emph{anytime-valid} sequential inference, even in composite and nonparametric settings. Our central contribution is to show that nonnegative martingale…
We consider the problem of distribution-free conformal prediction and the criterion of group conditional validity. This criterion is motivated by many practical scenarios including hidden stratification and group fairness. Existing methods achieve such guarantees under either restrictive grouping structure or distributional assumptions, or they are overly-conservative under heteroskedastic noise. We propose a simple reduction to the problem of achieving validity guarantees for individual populations by leveraging algorithms for a problem called multi-group learning. This allows us to port theo…
CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency
Komaba Institute for Science, Graduate School of Arts and Sciences, The University of Tokyo（东京大学艺术科学研究生院Komaba研究所）；Nagoya University（名古屋大学）；NARA Institute of Science and Technology (NAIST) / Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（NAIST科学与技术研究所 / 摩洛哥本·泽德人工智能大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) / RIKEN AIP（摩洛哥本·泽德人工智能大学 / RIKEN AIP）；Graduate School of Arts and Sciences, The University of Tokyo / RIKEN AIP / Kyoto University（东京大学艺术科学研究生院 / RIKEN AIP / 京都大学）
From Conditional to Unconditional Independence: Testing Conditional Independence via Transport Maps
Institute of Statistics and Big Data, Renmin University of China（中国人民大学统计与大数据研究院）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（香港理工大学数据科学与人工智能学院）；Department of Applied Mathematics, The Hong Kong Polytechnic University（香港理工大学应用数学系）
