With rapid adoption of deep learning in critical applications, the question of when and how much to trust these models often arises, which drives the need to quantify the inherent uncertainties. While identifying all sources that account for the stochasticity of models is challenging, it is common to augment predictions with confidence intervals to convey the expected variations in a model's behavior. We require prediction intervals to be well-calibrated, reflect the true uncertainties, and to be sharp. However, existing techniques for obtaining prediction intervals are known to produce unsatisfactory results in at least one of these criteria. To address this challenge, we develop a novel approach for building calibrated estimators. More specifically, we use separate models for prediction and interval estimation, and pose a bi-level optimization problem that allows the former to leverage estimates from the latter through an \textit{uncertainty matching} strategy. Using experiments in regression, time-series forecasting, and object localization, we show that our approach achieves significant improvements over existing uncertainty quantification methods, both in terms of model fidelity and calibration error.
In this paper, we propose a confidence-calibration method for predicting the winner of a famous multiplayer online battle arena (MOBA) game, League of Legends. In MOBA games, the dataset may contain a large amount of input-dependent noise; not all of such noise is observable. Hence, it is desirable to attempt a confidence-calibrated prediction. Unfortunately, most existing confidence calibration methods are pertaining to image and document classification tasks where consideration on uncertainty is not crucial. In this paper, we propose a novel calibration method that takes data uncertainty int…
Truthful Calibration Errors for Multi-Class Prediction
Peking University（北京大学）；Northwestern University（西北大学）；Microsoft Research, New England（微软研究院（新英格兰））；Northeastern University（东北大学）；Khoury College of Computer Sciences（计算机科学学院）
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Google Research（谷歌研究院）；MosaicML
Shortcut learning, in which models make use of easy-to-represent but unstable associations, is a major failure mode for robust machine learning. We study a flexible, causally-motivated approach to training robust predictors by discouraging the use of specific shortcuts, focusing on a common setting where a robust predictor could achieve optimal \emph{iid} generalization in principle, but is overshadowed by a shortcut predictor in practice. Our approach uses auxiliary labels, typically available at training time, to enforce conditional independences implied by the causal graph. We show both the…
Algorithms with Calibrated Machine Learning Predictions
Department of Management Science \& Engineering, Stanford University, Stanford, CA, USA；Department of Computer Science, Stanford University, Stanford, CA, USA
Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources
Hong Kong Baptist University（香港浸会大学）；Xidian University（西安电子科技大学）；University of Technology Sydney（悉尼科技大学）；The University of Sydney（悉尼大学）；The University of Melbourne（墨尔本大学）
This paper continues study, both theoretical and empirical, of the method of Venn prediction, concentrating on binary prediction problems. Venn predictors produce probability-type predictions for the labels of test objects which are guaranteed to be well calibrated under the standard assumption that the observations are generated independently from the same distribution. We give a simple formalization and proof of this property. We also introduce Venn-Abers predictors, a new class of Venn predictors based on the idea of isotonic regression, and report promising empirical results both for Venn-…
Learning to Augment Distributions for Out-of-Distribution Detection
Hong Kong Baptist University（香港浸会大学）；University of Technology Sydney（悉尼科技大学）；The University of Melbourne（墨尔本大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Predictors map individual instances in a population to the interval $[0,1]$. For a collection $\mathcal C$ of subsets of a population, a predictor is multi-calibrated with respect to $\mathcal C$ if it is simultaneously calibrated on each set in $\mathcal C$. We initiate the study of the construction of scaffolding sets, a small collection $\mathcal S$ of sets with the property that multi-calibration with respect to $\mathcal S$ ensures correctness, and not just calibration, of the predictor. Our approach is inspired by the folk wisdom that the intermediate layers of a neural net learn a highl…
Université Pierre et Marie Curie - Paris 6（巴黎第六大学（皮埃尔和玛丽·居里大学））；CNRS（法国国家科学研究中心）
Calibrated strategies can be obtained by performing strategies that have no internal regret in some auxiliary game. Such strategies can be constructed explicitly with the use of Blackwell's approachability theorem, in an other auxiliary game. We establish the converse: a strategy that approaches a convex $B$-set can be derived from the construction of a calibrated strategy. We develop these tools in the framework of a game with partial monitoring, where players do not observe the actions of their opponents but receive random signals, to define a notion of internal regret and construct strategi…
To Trust or Not to Trust: On Calibration in ML-based Resource Allocation for Wireless Networks
Centre for Wireless Innovation, Queen’s University of Belfast（女王大学贝尔法斯特无线创新中心）；Dhali Holdings Ltd.（Dhali控股有限公司）；School of Electrical and Computer Engineering, University of Campinas（坎皮纳斯大学电气与计算机工程学院）；Faculty of Engineering and Applied Science, Memorial University（纪念大学工程与应用科学学院）；School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast（女王大学贝尔法斯特电子、电气工程与计算机科学学院）
Conventional frequentist learning is known to yield poorly calibrated models that fail to reliably quantify the uncertainty of their decisions. Bayesian learning can improve calibration, but formal guarantees apply only under restrictive assumptions about correct model specification. Conformal prediction (CP) offers a general framework for the design of set predictors with calibration guarantees that hold regardless of the underlying data generation mechanism. However, when training data are limited, CP tends to produce large, and hence uninformative, predicted sets. This paper introduces a no…
University of California, Los Angeles（加州大学洛杉矶分校）；University of Cambridge（剑桥大学）；The Alan Turing Institute（艾伦图灵研究所）
Regularization improves generalization of supervised models to out-of-sample data. Prior works have shown that prediction in the causal direction (effect from cause) results in lower testing error than the anti-causal direction. However, existing regularization methods are agnostic of causality. We introduce Causal Structure Learning (CASTLE) regularization and propose to regularize a neural network by jointly learning the causal relationships between variables. CASTLE learns the causal directed acyclical graph (DAG) as an adjacency matrix embedded in the neural network's input layers, thereby…
Sari Agricultural Sciences and Natural Resources University（萨里农业科学与自然资源大学）；University of Tübingen（蒂宾根大学）；Ardakan University（阿尔达坎大学）；Mazandaran Agricultural and Natural Resources Research and Education Center（马赞德兰农业与自然资源研究教育中心）；Technische Universität Dresden（德累斯顿工业大学）；Oxford Brookes University（牛津布鲁克斯大学）；J. Selye University（哲利耶大学）
Estimation of the soil organic carbon content is of utmost importance in understanding the chemical, physical, and biological functions of the soil. This study proposes machine learning algorithms of support vector machines, artificial neural networks, regression tree, random forest, extreme gradient boosting, and conventional deep neural network for advancing prediction models of SOC. Models are trained with 1879 composite surface soil samples, and 105 auxiliary data as predictors. The genetic algorithm is used as a feature selection approach to identify effective variables. The results indic…
This paper considers the estimation and prediction of a high-dimensional linear regression in the setting of transfer learning, using samples from the target model as well as auxiliary samples from different but possibly related regression models. When the set of "informative" auxiliary samples is known, an estimator and a predictor are proposed and their optimality is established. The optimal rates of convergence for prediction and estimation are faster than the corresponding rates without using the auxiliary samples. This implies that knowledge from the informative auxiliary samples can be t…
University of Exeter（埃克塞特大学）；University of Glasgow（格拉斯哥大学）
For geospatial modelling and mapping tasks, variants of kriging - the spatial interpolation technique developed by South African mining engineer Danie Krige - have long been regarded as the established geostatistical methods. However, kriging and its variants (such as regression kriging, in which auxiliary variables or derivatives of these are included as covariates) are relatively restrictive models and lack capabilities that have been afforded to us in the last decade by deep neural networks. Principal among these is feature learning - the ability to learn filters to recognise task-specific …
