Out-of-distribution (OOD) data poses serious challenges in deployed machine learning models as even subtle changes could incur significant performance drops. Being able to estimate a model's performance on test data is important in practice as it indicates when to trust to model's decisions. We present a simple yet effective method to predict a model's performance on an unknown distribution without any addition annotation. Our approach is rooted in the Optimal Transport theory, viewing test samples' output softmax scores from deep neural networks as empirical samples from an unknown distribution. We show that our method, Confidence Optimal Transport (COT), provides robust estimates of a model's performance on a target domain. Despite its simplicity, our method achieves state-of-the-art results on three benchmark datasets and outperforms existing methods by a large margin.
UC Berkeley（加州大学伯克利分校）；Amazon（亚马逊）；Toyota Research Institute（丰田研究所）
Recent work has shown that the performance of machine learning models can vary substantially when models are evaluated on data drawn from a distribution that is close to but different from the training distribution. As a result, predicting model performance on unseen distributions is an important challenge. Our work connects techniques from domain adaptation and predictive uncertainty literature, and allows us to predict model accuracy on challenging unseen distributions without access to labeled data. In the context of distribution shift, distributional distances are often used to adapt model…
In this work, a machine learning approach is developed for predicting the outcomes of football matches. The novelty of this research lies in the utilisation of the Kelly Index to first classify matches into categories where each one denotes the different levels of predictive difficulty. Classification models using a wide suite of algorithms were developed for each category of matches in order to determine the efficacy of the approach. In conjunction to this, a set of previously unexplored features were engineering including Elo-based variables. The dataset originated from the Premier League ma…
Tsinghua University（清华大学）；University of Cambridge（剑桥大学）
It is often remarked that neural networks fail to increase their uncertainty when predicting on data far from the training distribution. Yet naively using softmax confidence as a proxy for uncertainty achieves modest success in tasks exclusively testing for this, e.g., out-of-distribution (OOD) detection. This paper investigates this contradiction, identifying two implicit biases that do encourage softmax confidence to correlate with epistemic uncertainty: 1) Approximately optimal decision boundary structure, and 2) Filtering effects of deep networks. It describes why low-dimensional intuition…
In this paper, we propose a confidence-calibration method for predicting the winner of a famous multiplayer online battle arena (MOBA) game, League of Legends. In MOBA games, the dataset may contain a large amount of input-dependent noise; not all of such noise is observable. Hence, it is desirable to attempt a confidence-calibrated prediction. Unfortunately, most existing confidence calibration methods are pertaining to image and document classification tasks where consideration on uncertainty is not crucial. In this paper, we propose a novel calibration method that takes data uncertainty int…
Optimal Transport-Induced Samples against Out-of-Distribution Overconfidence
Guangzhou University（广州大学）；SmartMore Corporation（SmartMore公司）；University of Science and Technology of China（中国科学技术大学）；Northwestern Polytechnical University（西北工业大学）；Guangdong Key Laboratory of Industrial Control System Security（广东工业控制系统安全重点实验室）
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples for which model confidence exceeds that threshold. ATC outperforms previous methods across several model architectures…
University of Guelph（圭尔夫大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Canadian Institute for Advanced Research（加拿大高级研究所）
It has been suggested that adversarial examples cause deep learning models to make incorrect predictions with high confidence. In this work, we take the opposite stance: an overly confident model is more likely to be vulnerable to adversarial examples. This work is one of the most proactive approaches taken to date, as we link robustness with non-calibrated model confidence on noisy images, providing a data-augmentation-free path forward. The adversarial examples phenomenon is most easily explained by the trend of increasing non-regularized model capacity, while the diversity and number of sam…
Carnegie Mellon University（卡内基梅隆大学）；Bosch Center for AI（博世人工智能中心）
Predicting how distributions over discrete variables vary over time is a common task in time series forecasting. But whereas most approaches focus on merely predicting the distribution at subsequent time steps, a crucial piece of information in many settings is to determine how this probability mass flows between the different elements over time. We propose a new approach to predicting such mass flow over time using optimal transport. Specifically, we propose a generic approach to predicting transport matrices in end-to-end deep learning systems, replacing the standard softmax operation with S…
Center for Data Anaytics and Biomedical Informatics, Temple University（天普大学数据分析与生物医学信息学中心）
For highly sensitive real-world predictive analytic applications such as healthcare and medicine, having good prediction accuracy alone is often not enough. These kinds of applications require a decision making process which uses uncertainty estimation as input whenever possible. Quality of uncertainty estimation is a subject of over or under confident prediction, which is often not addressed in many models. In this paper we show several extensions to the Gaussian Conditional Random Fields model, which aim to provide higher quality uncertainty estimation. These extensions are applied to the te…
CloneOT: predicting clone-specific perturbation responses with lineage-supervised neural inverse optimal transport
University of Maryland（马里兰大学）；Center of Bioinformatics and Computational Biology（生物信息学与计算生物学中心）；University of Pittsburgh（匹兹堡大学）；UPMC Children’s Hospital of Pittsburgh（匹兹堡UPMC儿童医院）；Institute of Health Computing（健康计算研究所）
Shahrekord University（沙赫尔库尔德大学）；Semnan University（塞姆南大学）；Obuda University（欧布达大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Slovak University of Technology in Bratislava（布拉迪斯拉发斯洛伐克工业大学）；Universiti Tenaga Nasional (UNITEN)（国家能源大学（UNITEN））；United Arab Emirates University（阿联酋大学）；University of Malaya (UM)（马来亚大学（UM））
Planting vegetation is one of the practical solutions for reducing sediment transfer rates. Increasing vegetation cover decreases environmental pollution and sediment transport rate (STR). Since sediments and vegetation interact complexly, predicting sediment transport rates is challenging. This study aims to predict sediment transport rate under vegetation cover using new and optimized versions of the group method of data handling (GMDH). Additionally, this study introduces a new ensemble model for predicting sediment transport rates. Model inputs include wave height, wave velocity, density c…
Wasserstein Parallel Transport for Predicting the Dynamics of Statistical Systems
Department of Statistics and Data Science, Carnegie Mellon University（卡内基梅隆大学统计与数据科学系）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）；Department of Economics, Emory University（埃默里大学经济系）
Bayesian sequence prediction is a simple technique for predicting future symbols sampled from an unknown measure on infinite sequences over a countable alphabet. While strong bounds on the expected cumulative error are known, there are only limited results on the distribution of this error. We prove tight high-probability bounds on the cumulative error, which is measured in terms of the Kullback-Leibler (KL) divergence. We also consider the problem of constructing upper confidence bounds on the KL and Hellinger errors similar to those constructed from Hoeffding-like bounds in the i.i.d. case. …
Technion - Israel Institute of Technology（以色列理工学院）；Toyoyta Technological Institute at Chicago（丰田工业大学芝加哥分校）
Machine learning plays an increasingly significant role in many aspects of our lives (including medicine, transportation, security, justice and other domains), making the potential consequences of false predictions increasingly devastating. These consequences may be mitigated if we can automatically flag such false predictions and potentially assign them to alternative, more reliable mechanisms, that are possibly more costly and involve human attention. This suggests the task of detecting errors, which we tackle in this paper for the case of visual classification. To this end, we propose a nov…
AWS AI Labs（AWS人工智能实验室）；University of Southern California（南加州大学）
We propose an approach to estimate the number of samples required for a model to reach a target performance. We find that the power law, the de facto principle to estimate model performance, leads to large error when using a small dataset (e.g., 5 samples per class) for extrapolation. This is because the log-performance error against the log-dataset size follows a nonlinear progression in the few-shot regime followed by a linear progression in the high-shot regime. We introduce a novel piecewise power law (PPL) that handles the two data regimes differently. To estimate the parameters of the PP…
Université de Montréal（蒙特利尔大学）；Polytechnique Montréal（蒙特利尔综合理工学院）
An important aspect of the quality of a public transport service is its reliability, which is defined as the invariability of the service attributes. Preventive measures taken during planning can reduce risks of unreliability throughout operations. In order to tackle reliability during the service planning phase, a key piece of information is the long-term prediction of the density of the travel time, which conveys the uncertainty of travel times. We introduce a reliable approach to one of the problems of service planning in public transport, namely the Multiple Depot Vehicle Scheduling Proble…
Predicting Critical Heat Flux with Uncertainty Quantification and Domain Generalization Using Conditional Variational Autoencoders and Deep Neural Networks
IMT Atlantique（大西洋高等电信学校）；Lab-STICC（布列塔尼电子与电信通信科学实验室）；MILA, Université de Montréal（蒙特利尔大学MILA人工智能研究所）；Université Cote d’Azur（蔚蓝海岸大学）；École Normale Supérieure de Lyon（里昂高等师范学院）
In the context of few-shot learning, one cannot measure the generalization ability of a trained classifier using validation sets, due to the small number of labeled samples. In this paper, we are interested in finding alternatives to answer the question: is my classifier generalizing well to previously unseen data? We first analyze the reasons for the variability of generalization performances. We then investigate the case of using transfer-based solutions, and consider three settings: i) supervised where we only have access to a few labeled samples, ii) semi-supervised where we have access to…
Cyclica（赛克利卡公司）；University of Toronto（多伦多大学）；Princess Margaret Cancer Centre（玛格丽特公主癌症中心）；University Health Network（大学健康网络）；Vector Institute（向量研究所）
Performance of neural network models relies on the availability of large datasets with minimal levels of uncertainty. Transfer Learning (TL) models have been proposed to resolve the issue of small dataset size by letting the model train on a bigger, task-related reference dataset and then fine-tune on a smaller, task-specific dataset. In this work, we apply a transfer learning approach to improve predictive power in noisy data systems with large variable confidence datasets. We propose a deep neural network method called Filtered Transfer Learning (FTL) that defines multiple tiers of data conf…
This work provides two statistical Gaussian forecasting methods for predicting First Daily Departure Times (FDDTs) of everyday use electric vehicles. This is important in smart grid applications to understand disconnection times of such mobile storage units, for instance to forecast storage of non dispatchable loads (e.g. wind and solar power). We provide a review of the relevant state-of-the-art driving behavior features towards FDDT prediction, to then propose an approximated Gaussian method which qualitatively forecasts how many vehicles will depart within a given time frame, by assuming th…
Accurately forecasting transportation demand is crucial for efficient urban traffic guidance, control and management. One solution to enhance the level of prediction accuracy is to leverage graph convolutional networks (GCN), a neural network based modelling approach with the ability to process data contained in graph based structures. As a powerful extension of GCN, a spatial-temporal graph convolutional network (ST-GCN) aims to capture the relationship of data contained in the graphical nodes across both spatial and temporal dimensions, which presents a novel deep learning paradigm for the a…
PRESCRIBE: Predicting Single-Cell Responses with Bayesian Estimation
Shanghai Jiao Tong University（上海交通大学）；Westlake University（西湖大学）；AIMS Lab, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）AIMS实验室）；The Hong Kong University of Science and Technology（香港科技大学）
