Quantifying uncertainty in model predictions is a common goal for practitioners seeking more than just point predictions. One tool for uncertainty quantification that requires minimal assumptions is conformal inference, which can help create probabilistically valid prediction regions for black box models. Classical conformal prediction only provides marginal validity, whereas in many situations locally valid prediction regions are desirable. Deciding how best to partition the feature space X when applying localized conformal prediction is still an open question. We present MD-split+, a practical local conformal approach that creates X partitions based on localized model performance of conditional density estimation models. Our method handles complex real-world data settings where such models may be misspecified, and scales to high-dimensional inputs. We discuss how our local partitions philosophically align with expected behavior from an unattainable conditional conformal inference approach. We also empirically compare our method against other local conformal approaches.
University of Oxford（牛津大学）；St. Anne’s College（圣安妮学院）
Modern black-box predictive models are often accompanied by weak performance guarantees that only hold asymptotically in the size of the dataset or require strong parametric assumptions. In response to this, split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions. Although prediction set validity most often concerns marginal coverage, we explore the related but different guarantee of tolerance regions, reformulating known results in the language of nested prediction sets and extending on the duality between margina…
Split Conformal Prediction under Data Contamination
Department of Statistics, University of Oxford（统计系，牛津大学）；Department of Statistics, University of Warwick（统计系，沃里克大学）；Department of Mathematics, UCLA（数学系，加州大学洛杉矶分校）；Department of Mathematics, Université libre de Bruxelles（数学系，布鲁塞尔自由大学）
Conformal prediction is a simple and powerful tool that can quantify uncertainty without any distributional assumptions. Many existing methods only address the average coverage guarantee, which is not ideal compared to the stronger conditional coverage guarantee. Existing methods of approximating conditional coverage require additional models or time effort, which makes them not easy to scale. In this paper, we propose a modified non-conformity score by leveraging the local approximation of the conditional distribution using kernel density estimation. The modified score inherits the spirit of …
Multi-Variable Conformal Prediction: Optimizing Prediction Sets without Data Splitting
TU Munich & MCML, Germany（慕尼黑技术大学及MCML，德国）；Politecnico di Milano, Italy（米兰理工学院，意大利）；University of Brescia, Italy（布雷西亚大学，意大利）；ETH Zürich, Switzerland（苏黎世联邦理工学院，瑞士）
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive systems; the input predictive system is not supposed to satisfy any properties of validity, whereas the output predictive system is guaranteed to be calibrated in probability. It is interesting that the …
On Optimal Data Splitting for Split Conformal Prediction
Department of Statistics and Data Science, Washington University in St. Louis（华盛顿大学圣路易斯分校统计与数据科学系）；Department of Electrical and Systems Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校电气与系统工程系）
Split conformal prediction is a computationally efficient method for performing distribution-free predictive inference in regression. It involves, however, a one-time random split of the data, and the result depends on the particular split. To address this problem, we propose multi split conformal prediction, a simple method based on Markov's inequality to aggregate single split conformal prediction intervals across multiple splits.
School of Mathematics and Statistics（数学与统计学学院）；Beijing Jiaotong University（北京交通大学）；Department of Mathematical and Statistical Sciences（数学与统计学系）；University of Alberta（阿尔伯塔大学）
Conformal prediction for frequency-severity modeling
School of Applied Mathematics, Getulio Vargas Foundation（应用数学学院，格尔瓦斯基金会）；Insper Institute of Education and Research（Insper教育与研究学院）；SUSEP - Superintendence of Private Insurance（私人保险监管局）；UERJ - State University of Rio de Janeiro（里约热内卢州立大学）
Predictive inference for time series: why is split conformal effective despite temporal dependence?
Department of Statistics, University of Chicago（芝加哥大学统计系）；Schools of Industrial and Systems Engineering and Electrical and Computer Engineering, Georgia Tech（佐治亚理工学院工业与系统工程及电气与计算机工程学院）
LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees
Federal University of São Carlos（巴西圣卡洛斯联邦大学）；University of São Paulo（圣保罗大学）；Institute of Mathematics and Computer Science（数学与计算机科学学院）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化技术研究院）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立应用科学研究院）；LJK（格勒诺布尔联合实验室）
In this work, we study some novel applications of conformal inference techniques to the problem of providing machine learning procedures with more transparent, accurate, and practical performance guarantees. We provide a natural extension of the traditional conformal prediction framework, done in such a way that we can make valid and well-calibrated predictive statements about the future performance of arbitrary learning algorithms, when passed an as-yet unseen training set. In addition, we include some nascent empirical examples to illustrate potential applications.
Building on top of a regression model, Conformal Prediction methods produce distribution free prediction sets, requiring only i.i.d. data. While R packages implementing such methods for the univariate response framework have been developed, this is not the case with multivariate and functional responses. conformalInference.multi and conformalInference.fd address this void, by extending classical and more advanced conformal prediction methods like full conformal, split conformal, jackknife+ and multi split conformal to deal with the multivariate and functional case. The extreme flexibility of c…
Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry
Epistemic Intelligence & Computation Lab, College of Computing & Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算与数据科学学院）；CISPA Helmholtz Center for Information Security, Germany（德国亥姆霍兹信息安全中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Institute of Informatics, LMU Munich, Germany（德国慕尼黑机器学习中心）；Munich Center for Machine Learning, Germany（曼彻斯特人工智能基础研究中心）；Manchester Centre for AI Fundamentals, Manchester, United Kingdom Department of Computer Science, The University of Manchester, United Kingdom
We prove calibration guarantees for the popular histogram binning (also called uniform-mass binning) method of Zadrozny and Elkan [2001]. Histogram binning has displayed strong practical performance, but theoretical guarantees have only been shown for sample split versions that avoid 'double dipping' the data. We demonstrate that the statistical cost of sample splitting is practically significant on a credit default dataset. We then prove calibration guarantees for the original method that double dips the data, using a certain Markov property of order statistics. Based on our results, we make …
University of Southern California（南加州大学）；Zhejiang University（浙江大学）
This paper develops novel conformal methods to test whether a new observation was sampled from the same distribution as a reference set. Blending inductive and transductive conformal inference in an innovative way, the described methods can re-weight standard conformal p-values based on dependent side information from known out-of-distribution data in a principled way, and can automatically take advantage of the most powerful model from any collection of one-class and binary classifiers. The solution can be implemented either through sample splitting or via a novel transductive cross-validatio…
Department of Computer Science, New Jersey Institute of Technology（计算机科学系，新泽西理工学院）；Department of Mathematical Sciences, New Jersey Institute of Technology（数学科学系，新泽西理工学院）
Moment-Calibrated Conformal Prediction with Side Information
School of Statistics and Data Science, Nankai University（南开大学统计与数据科学学院）；School of Mathematics and Statistics, University of Melbourne（墨尔本大学数学与统计学院）；School of Mathematical Science, Chengdu University of Technology（成都理工大学数学科学学院）
Conformal methods create prediction bands that control average coverage assuming solely i.i.d. data. Although the literature has mostly focused on prediction intervals, more general regions can often better represent uncertainty. For instance, a bimodal target is better represented by the union of two intervals. Such prediction regions are obtained by CD-split , which combines the split method and a data-driven partition of the feature space which scales to high dimensions. CD-split however contains many tuning parameters, and their role is not clear. In this paper, we provide new insights on …
