Predictive inference for time series: why is split conformal effective despite temporal dependence?
Department of Statistics, University of Chicago（芝加哥大学统计系）；Schools of Industrial and Systems Engineering and Electrical and Computer Engineering, Georgia Tech（佐治亚理工学院工业与系统工程及电气与计算机工程学院）
We consider the problem of uncertainty quantification for prediction in a time series: if we use past data to forecast the next time point, can we provide valid prediction intervals around our forecasts? To avoid placing distributional assumptions on the data, in recent years the conformal prediction method has been a popular approach for predictive inference, since it provides distribution-free coverage for any iid or exchangeable data distribution. However, in the time series setting, the strong empirical performance of conformal prediction methods is not well understood, since even short-range temporal dependence is a strong violation of the exchangeability assumption. Using predictors with "memory" -- i.e., predictors that utilize past observations, such as autoregressive models -- further exacerbates this problem. In this work, we examine the theoretical properties of split conformal prediction in the time series setting, including the case where predictors may have memory. Our results bound the loss of coverage of these methods in terms of a new "switch coefficient", measuring the extent to which temporal dependence within the time series creates violations of exchangeability. Our characterization of the coverage probability is sharp over the class of stationary, $β$-mixing processes. Along the way, we introduce tools that may prove useful in analyzing other predictive inference methods for dependent data.
On Optimal Data Splitting for Split Conformal Prediction
Department of Statistics and Data Science, Washington University in St. Louis（华盛顿大学圣路易斯分校统计与数据科学系）；Department of Electrical and Systems Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校电气与系统工程系）
Quantifying uncertainty in model predictions is a common goal for practitioners seeking more than just point predictions. One tool for uncertainty quantification that requires minimal assumptions is conformal inference, which can help create probabilistically valid prediction regions for black box models. Classical conformal prediction only provides marginal validity, whereas in many situations locally valid prediction regions are desirable. Deciding how best to partition the feature space X when applying localized conformal prediction is still an open question. We present MD-split+, a practic…
Massachusetts Institute of Technology（麻省理工学院）；BITS Pilani Goa Campus（BITS Pilani果阿校区）
Understanding and quantifying uncertainty in black box Neural Networks (NNs) is critical when deployed in real-world settings such as healthcare. Recent works using Bayesian and non-Bayesian methods have shown how a unified predictive uncertainty can be modelled for NNs. Decomposing this uncertainty to disentangle the granular sources of heteroscedasticity in data provides rich information about its underlying causes. We propose a conceptually simple non-Bayesian approach, deep split ensemble, to disentangle the predictive uncertainties using a multivariate Gaussian mixture model. The NNs are …
A Gentle Introduction to Conformal Time Series Forecasting
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Royal Holloway, University of London（伦敦大学皇家霍洛威学院）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫人工智能大学）；University of Mons（蒙斯大学）
We develop a distribution-free, unsupervised anomaly detection method called ECAD, which wraps around any regression algorithm and sequentially detects anomalies. Rooted in conformal prediction, ECAD does not require data exchangeability but approximately controls the Type-I error when data are normal. Computationally, it involves no data-splitting and efficiently trains ensemble predictors to increase statistical power. We demonstrate the superior performance of ECAD on detecting anomalous spatio-temporal traffic flow.
Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry
Epistemic Intelligence & Computation Lab, College of Computing & Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算与数据科学学院）；CISPA Helmholtz Center for Information Security, Germany（德国亥姆霍兹信息安全中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Institute of Informatics, LMU Munich, Germany（德国慕尼黑机器学习中心）；Munich Center for Machine Learning, Germany（曼彻斯特人工智能基础研究中心）；Manchester Centre for AI Fundamentals, Manchester, United Kingdom Department of Computer Science, The University of Manchester, United Kingdom
University of Oxford（牛津大学）；St. Anne’s College（圣安妮学院）
Modern black-box predictive models are often accompanied by weak performance guarantees that only hold asymptotically in the size of the dataset or require strong parametric assumptions. In response to this, split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions. Although prediction set validity most often concerns marginal coverage, we explore the related but different guarantee of tolerance regions, reformulating known results in the language of nested prediction sets and extending on the duality between margina…
Conformal methods create prediction bands that control average coverage assuming solely i.i.d. data. Although the literature has mostly focused on prediction intervals, more general regions can often better represent uncertainty. For instance, a bimodal target is better represented by the union of two intervals. Such prediction regions are obtained by CD-split , which combines the split method and a data-driven partition of the feature space which scales to high dimensions. CD-split however contains many tuning parameters, and their role is not clear. In this paper, we provide new insights on …
Split Conformal Prediction under Data Contamination
Department of Statistics, University of Oxford（统计系，牛津大学）；Department of Statistics, University of Warwick（统计系，沃里克大学）；Department of Mathematics, UCLA（数学系，加州大学洛杉矶分校）；Department of Mathematics, Université libre de Bruxelles（数学系，布鲁塞尔自由大学）
Conformal prediction is a simple and powerful tool that can quantify uncertainty without any distributional assumptions. Many existing methods only address the average coverage guarantee, which is not ideal compared to the stronger conditional coverage guarantee. Existing methods of approximating conditional coverage require additional models or time effort, which makes them not easy to scale. In this paper, we propose a modified non-conformity score by leveraging the local approximation of the conditional distribution using kernel density estimation. The modified score inherits the spirit of …
Multi-Variable Conformal Prediction: Optimizing Prediction Sets without Data Splitting
TU Munich & MCML, Germany（慕尼黑技术大学及MCML，德国）；Politecnico di Milano, Italy（米兰理工学院，意大利）；University of Brescia, Italy（布雷西亚大学，意大利）；ETH Zürich, Switzerland（苏黎世联邦理工学院，瑞士）
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive systems; the input predictive system is not supposed to satisfy any properties of validity, whereas the output predictive system is guaranteed to be calibrated in probability. It is interesting that the …
Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational inefficiency of conformal predictors. In this paper we discuss two computationally efficient versions of conformal predictive systems, which we call split conformal predictive systems and cross-conformal predictive syste…
We discuss a concept denoted as Conformal Prediction (CP) in this paper. While initially stemming from the world of machine learning, it was never applied or analyzed in the context of short-term electricity price forecasting. Therefore, we elaborate the aspects that render Conformal Prediction worthwhile to know and explain why its simple yet very efficient idea has worked in other fields of application and why its characteristics are promising for short-term power applications as well. We compare its performance with different state-of-the-art electricity price forecasting models such as qua…
School of Mathematics and Statistics（数学与统计学学院）；Beijing Jiaotong University（北京交通大学）；Department of Mathematical and Statistical Sciences（数学与统计学系）；University of Alberta（阿尔伯塔大学）
Conformal prediction for frequency-severity modeling
School of Applied Mathematics, Getulio Vargas Foundation（应用数学学院，格尔瓦斯基金会）；Insper Institute of Education and Research（Insper教育与研究学院）；SUSEP - Superintendence of Private Insurance（私人保险监管局）；UERJ - State University of Rio de Janeiro（里约热内卢州立大学）
We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conform…
LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees
Federal University of São Carlos（巴西圣卡洛斯联邦大学）；University of São Paulo（圣保罗大学）；Institute of Mathematics and Computer Science（数学与计算机科学学院）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化技术研究院）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立应用科学研究院）；LJK（格勒诺布尔联合实验室）
Conformal methods create prediction bands that control average coverage under no assumptions besides i.i.d. data. Besides average coverage, one might also desire to control conditional coverage, that is, coverage for every new testing point. However, without strong assumptions, conditional coverage is unachievable. Given this limitation, the literature has focused on methods with asymptotical conditional coverage. In order to obtain this property, these methods require strong conditions on the dependence between the target variable and the features. We introduce two conformal methods based on …
