Conformal methods create prediction bands that control average coverage assuming solely i.i.d. data. Although the literature has mostly focused on prediction intervals, more general regions can often better represent uncertainty. For instance, a bimodal target is better represented by the union of two intervals. Such prediction regions are obtained by CD-split , which combines the split method and a data-driven partition of the feature space which scales to high dimensions. CD-split however contains many tuning parameters, and their role is not clear. In this paper, we provide new insights on CD-split by exploring its theoretical properties. In particular, we show that CD-split converges asymptotically to the oracle highest predictive density set and satisfies local and asymptotic conditional validity. We also present simulations that show how to tune CD-split. Finally, we introduce HPD-split, a variation of CD-split that requires less tuning, and show that it shares the same theoretical guarantees as CD-split. In a wide variety of our simulations, CD-split and HPD-split have better conditional coverage and yield smaller prediction regions than other methods.
CP4SBI: Local Conformal Calibration of Credible Sets in Simulation-Based Inference
Department of Statistics, Federal University of São Carlos（统计系，圣卡洛斯联邦大学）；Institute of Mathematics and Computer Science, University of São Paulo（数学与计算机科学学院，圣保罗大学）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学，法国国家信息与自动化研究所，法国国家科学研究中心，格勒诺布尔INP，LJK）
Quantifying uncertainty in model predictions is a common goal for practitioners seeking more than just point predictions. One tool for uncertainty quantification that requires minimal assumptions is conformal inference, which can help create probabilistically valid prediction regions for black box models. Classical conformal prediction only provides marginal validity, whereas in many situations locally valid prediction regions are desirable. Deciding how best to partition the feature space X when applying localized conformal prediction is still an open question. We present MD-split+, a practic…
University of Oxford（牛津大学）；St. Anne’s College（圣安妮学院）
Modern black-box predictive models are often accompanied by weak performance guarantees that only hold asymptotically in the size of the dataset or require strong parametric assumptions. In response to this, split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions. Although prediction set validity most often concerns marginal coverage, we explore the related but different guarantee of tolerance regions, reformulating known results in the language of nested prediction sets and extending on the duality between margina…
Split Conformal Prediction under Data Contamination
Department of Statistics, University of Oxford（统计系，牛津大学）；Department of Statistics, University of Warwick（统计系，沃里克大学）；Department of Mathematics, UCLA（数学系，加州大学洛杉矶分校）；Department of Mathematics, Université libre de Bruxelles（数学系，布鲁塞尔自由大学）
Conformal prediction is a simple and powerful tool that can quantify uncertainty without any distributional assumptions. Many existing methods only address the average coverage guarantee, which is not ideal compared to the stronger conditional coverage guarantee. Existing methods of approximating conditional coverage require additional models or time effort, which makes them not easy to scale. In this paper, we propose a modified non-conformity score by leveraging the local approximation of the conditional distribution using kernel density estimation. The modified score inherits the spirit of …
Multi-Variable Conformal Prediction: Optimizing Prediction Sets without Data Splitting
TU Munich & MCML, Germany（慕尼黑技术大学及MCML，德国）；Politecnico di Milano, Italy（米兰理工学院，意大利）；University of Brescia, Italy（布雷西亚大学，意大利）；ETH Zürich, Switzerland（苏黎世联邦理工学院，瑞士）
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive systems; the input predictive system is not supposed to satisfy any properties of validity, whereas the output predictive system is guaranteed to be calibrated in probability. It is interesting that the …
Conformal prediction for frequency-severity modeling
School of Applied Mathematics, Getulio Vargas Foundation（应用数学学院，格尔瓦斯基金会）；Insper Institute of Education and Research（Insper教育与研究学院）；SUSEP - Superintendence of Private Insurance（私人保险监管局）；UERJ - State University of Rio de Janeiro（里约热内卢州立大学）
On Optimal Data Splitting for Split Conformal Prediction
Department of Statistics and Data Science, Washington University in St. Louis（华盛顿大学圣路易斯分校统计与数据科学系）；Department of Electrical and Systems Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校电气与系统工程系）
Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational inefficiency of conformal predictors. In this paper we discuss two computationally efficient versions of conformal predictive systems, which we call split conformal predictive systems and cross-conformal predictive syste…
School of Mathematics and Statistics（数学与统计学学院）；Beijing Jiaotong University（北京交通大学）；Department of Mathematical and Statistical Sciences（数学与统计学系）；University of Alberta（阿尔伯塔大学）
Conformal methods create prediction bands that control average coverage under no assumptions besides i.i.d. data. Besides average coverage, one might also desire to control conditional coverage, that is, coverage for every new testing point. However, without strong assumptions, conditional coverage is unachievable. Given this limitation, the literature has focused on methods with asymptotical conditional coverage. In order to obtain this property, these methods require strong conditions on the dependence between the target variable and the features. We introduce two conformal methods based on …
Predictive inference for time series: why is split conformal effective despite temporal dependence?
Department of Statistics, University of Chicago（芝加哥大学统计系）；Schools of Industrial and Systems Engineering and Electrical and Computer Engineering, Georgia Tech（佐治亚理工学院工业与系统工程及电气与计算机工程学院）
We develop a distribution-free, unsupervised anomaly detection method called ECAD, which wraps around any regression algorithm and sequentially detects anomalies. Rooted in conformal prediction, ECAD does not require data exchangeability but approximately controls the Type-I error when data are normal. Computationally, it involves no data-splitting and efficiently trains ensemble predictors to increase statistical power. We demonstrate the superior performance of ECAD on detecting anomalous spatio-temporal traffic flow.
LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees
Federal University of São Carlos（巴西圣卡洛斯联邦大学）；University of São Paulo（圣保罗大学）；Institute of Mathematics and Computer Science（数学与计算机科学学院）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化技术研究院）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立应用科学研究院）；LJK（格勒诺布尔联合实验室）
University of Southern California（南加州大学）；Zhejiang University（浙江大学）
This paper develops novel conformal methods to test whether a new observation was sampled from the same distribution as a reference set. Blending inductive and transductive conformal inference in an innovative way, the described methods can re-weight standard conformal p-values based on dependent side information from known out-of-distribution data in a principled way, and can automatically take advantage of the most powerful model from any collection of one-class and binary classifiers. The solution can be implemented either through sample splitting or via a novel transductive cross-validatio…
We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conform…
Leave a Window Out: Modifying the Jackknife for Predictive Inference in Time Series
Schools of Industrial and Systems Engineering and Electrical and Computer Engineering（工业与系统工程系和电气与计算机工程系）；Department of Statistics, University of Chicago（芝加哥大学统计系）
Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity
Department of Environment, Ghent University（根特大学环境系）；Institute of Informatics, LMU Munich（慕尼黑大学信息学研究所）；MCML, Munich（慕尼黑机器学习中心）；DFKI (DSA), Kaiserslautern（德国人工智能研究中心（数据分析与智能系统部门），凯泽斯劳滕）；Dept. of Data Analysis and Mathematical Modelling, Ghent University（根特大学数据分析与数学建模系）
Conformal Prediction Intervals with Tail-Specific Guarantees
Sapienza University of Rome（罗马第一大学）；MEMOTEF Department, Sapienza University of Rome（罗马第一大学MEMOTEF系）；MRC–Biostatistics Unit, University of Cambridge（剑桥大学医学研究委员会生物统计单元）
