Considering uncertainty estimation of modern neural networks (NNs) is one of the most important steps towards deploying machine learning systems to meaningful real-world applications such as in medicine, finance or autonomous systems. At the moment, ensembles of different NNs constitute the state-of-the-art in both accuracy and uncertainty estimation in different tasks. However, ensembles of NNs are unpractical under real-world constraints, since their computation and memory consumption scale linearly with the size of the ensemble, which increase their latency and deployment cost. In this work, we examine a simple regularisation approach for distribution-free knowledge distillation of ensemble of machine learning models into a single NN. The aim of the regularisation is to preserve the diversity, accuracy and uncertainty estimation characteristics of the original ensemble without any intricacies, such as fine-tuning. We demonstrate the generality of the approach on combinations of toy data, SVHN/CIFAR-10, simple to complex NN architectures and different tasks.
Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation
Graduate School of Data Science, Seoul National University（首尔国立大学数据科学研究生院）；Department of Rural Systems Engineering, Seoul National University（首尔国立大学农村系统工程系）；Department of Aerospace Engineering, Seoul National University（首尔国立大学航空航天工程系）；Department of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology（全州科学科技研究院电子工程与计算机科学系）
Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions
South China University of Technology（华南理工大学）；Tencent Financial Technology（腾讯金融科技）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Soochow University（苏州大学）；Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University（浙江省智能教育技术与应用重点实验室，浙江师范大学）
Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Department of Electrical Engineering Indian Institute of Technology Bombay 21D070002（电子工程系 印度理工学院班加罗尔 21D070002）；Department of Electrical Engineering Indian Institute of Technology Bombay 210010010（电子工程系 印度理工学院班加罗尔 210010010）；Department of Electrical Engineering Indian Institute of Technology Bombay 210020023（电子工程系 印度理工学院班加罗尔 210020023）
Yonsei University（延世大学）；Seoul National University（首尔大学）
Active learning can be defined as iterations of data labeling, model training, and data acquisition, until sufficient labels are acquired. A traditional view of data acquisition is that, through iterations, knowledge from human labels and models is implicitly distilled to monotonically increase the accuracy and label consistency. Under this assumption, the most recently trained model is a good surrogate for the current labeled data, from which data acquisition is requested based on uncertainty/diversity. Our contribution is debunking this myth and proposing a new objective for distillation. Fi…
Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad（计算机科学与工程系，印度理工学院海得拉巴分校）；Jagiellonian University, Faculty of Mathematics and Computer Science（雅盖隆大学，数学与计算机科学学院）
Current deep learning models often suffer from catastrophic forgetting of old knowledge when continually learning new knowledge. Existing strategies to alleviate this issue often fix the trade-off between keeping old knowledge (stability) and learning new knowledge (plasticity). However, the stability-plasticity trade-off during continual learning may need to be dynamically changed for better model performance. In this paper, we propose two novel ways to adaptively balance model stability and plasticity. The first one is to adaptively integrate multiple levels of old knowledge and transfer it …
Imperial College London（帝国理工学院）；University of Amsterdam（阿姆斯特丹大学）；ETH Zurich（苏黎世联邦理工学院）；University of Warsaw（华沙大学）；Google Research（谷歌研究院）；University of California, Irvine（加州大学欧文分校）
Ensembles of models have been empirically shown to improve predictive performance and to yield robust measures of uncertainty. However, they are expensive in computation and memory. Therefore, recent research has focused on distilling ensembles into a single compact model, reducing the computational and memory burden of the ensemble while trying to preserve its predictive behavior. Most existing distillation formulations summarize the ensemble by capturing its average predictions. As a result, the diversity of the ensemble predictions, stemming from each member, is lost. Thus, the distilled mo…
Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading
School of Public Health, Zhejiang University（浙江大学公共卫生学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Liangzhu Laboratory and WeDoctor Cloud（良渚实验室和WeDoctor云）；State Key Laboratory of Transvascular Implantation Devices, The Second Affiliated Hospital Zhejiang University School of Medicine（浙江大学医学院第二附属医院心血管植入器械国家重点实验室）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）
One of the major limitations of deep learning models is that they face catastrophic forgetting in an incremental learning scenario. There have been several approaches proposed to tackle the problem of incremental learning. Most of these methods are based on knowledge distillation and do not adequately utilize the information provided by older task models, such as uncertainty estimation in predictions. The predictive uncertainty provides the distributional information can be applied to mitigate catastrophic forgetting in a deep learning framework. In the proposed work, we consider a Bayesian fo…
Bayesian interpretations of neural network have a long history, dating back to early work in the 1990's and have recently regained attention because of their desirable properties like uncertainty estimation, model robustness and regularisation. We want to discuss here the application of Bayesian models to knowledge sharing between neural networks. Knowledge sharing comes in different facets, such as transfer learning, model distillation and shared embeddings. All of these tasks have in common that learned "features" ought to be shared across different networks. Theoretically rooted in the conc…
Zenith Ai UK（英国Zenith人工智能公司）；University of Oxford（牛津大学）；Queen’s University Belfast（贝尔法斯特女王大学）
To train robust deep neural networks (DNNs), we systematically study several target modification approaches, which include output regularisation, self and non-self label correction (LC). Two key issues are discovered: (1) Self LC is the most appealing as it exploits its own knowledge and requires no extra models. However, how to automatically decide the trust degree of a learner as training goes is not well answered in the literature? (2) Some methods penalise while the others reward low-entropy predictions, prompting us to ask which one is better? To resolve the first issue, taking two well-a…
Cross-Paradigm Knowledge Distillation: A Comprehensive Study of Bidirectional Transfer Between Random Forests and Deep Neural Networks for Big Data Applications
Rethinking Reverse KL as Adaptive Entropy Distillation
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Science and Technology, Hong Kong Metropolitan University（香港都会大学科技学院）
New Jersey Institute of Technology（新泽西理工学院）；Microsoft Research（微软研究院）；Rice University（莱斯大学）
Recent work has focused on compressing pre-trained language models (PLMs) like BERT where the major focus has been to improve the in-distribution performance for downstream tasks. However, very few of these studies have analyzed the impact of compression on the generalizability and robustness of compressed models for out-of-distribution (OOD) data. Towards this end, we study two popular model compression techniques including knowledge distillation and pruning and show that the compressed models are significantly less robust than their PLM counterparts on OOD test sets although they obtain simi…
University of Padova（帕多瓦大学）；University of Catania（卡塔尼亚大学）
Human capability to anticipate near future from visual observations and non-verbal cues is essential for developing intelligent systems that need to interact with people. Several research areas, such as human-robot interaction (HRI), assisted living or autonomous driving need to foresee future events to avoid crashes or help people. Egocentric scenarios are classic examples where action anticipation is applied due to their numerous applications. Such challenging task demands to capture and model domain's hidden structure to reduce prediction uncertainty. Since multiple actions may equally occu…
McGill University（麦吉尔大学）；Centre for Intelligent Machines (CIM)（智能机器中心 (CIM)）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Segmentation of enhancing tumours or lesions from MRI is important for detecting new disease activity in many clinical contexts. However, accurate segmentation requires the inclusion of medical images (e.g., T1 post contrast MRI) acquired after injecting patients with a contrast agent (e.g., Gadolinium), a process no longer thought to be safe. Although a number of modality-agnostic segmentation networks have been developed over the past few years, they have been met with limited success in the context of enhancing pathology segmentation. In this work, we present HAD-Net, a novel offline advers…
Knowledge Distillation and Enhanced Subdomain Adaptation Using Graph Convolutional Network for Resource-Constrained Bearing Fault Diagnosis
Department of Electrical and Computer Engineering, Tarbiat Modares University, Iran（电子与计算机工程系，塔里比亚特·莫达雷斯大学，伊朗）；Faculty of Engineering and Technology, University of Mazandaran, Iran（工程与技术学院，马赞德兰大学，伊朗）；Department of Medicine, Baylor College of Medicine, USA（医学系，贝勒医学院，美国）
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
ITS Lab, Institute of Computer Science, University of Tartu（塔尔图大学计算机科学学院ITS实验室）；LITIS, Université de Rouen（鲁昂大学LITIS实验室）；LITIS, INSA de Rouen（鲁昂国立应用科学学院LITIS实验室）
In this work we aim to obtain computationally-efficient uncertainty estimates with deep networks. For this, we propose a modified knowledge distillation procedure that achieves state-of-the-art uncertainty estimates both for in and out-of-distribution samples. Our contributions include a) demonstrating and adapting to distillation's regularization effect b) proposing a novel target teacher distribution c) a simple augmentation procedure to improve out-of-distribution uncertainty estimates d) shedding light on the distillation procedure through comprehensive set of experiments.
Deep learning is increasingly being applied in safety-critical domains. For these scenarios it is important to know the level of uncertainty in a model's prediction to ensure appropriate decisions are made by the system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles often significantly increase the resources required in the training and/or deployment phases. Approaches have been developed that typically address the costs in one of these phases. In this work we propose a novel training approach, self-distribution distillation …
Hong Kong University of Science and Technology（香港科技大学）
When recognizing emotions, subtle nuances in displays of emotion generate ambiguity or uncertainty in emotion perception. Emotion uncertainty has been previously interpreted as inter-rater disagreement among multiple annotators. In this paper, we consider a more common and challenging scenario: modeling emotion uncertainty when only single emotion labels are available. From a Bayesian perspective, we propose to use deep ensembles to capture uncertainty for multiple emotion descriptors, i.e., action units, discrete expression labels and continuous descriptors. We further apply iterative self-di…
Calibrated estimates of uncertainty are critical for many real-world computer vision applications of deep learning. While there are several widely-used uncertainty estimation methods, dropout inference stands out for its simplicity and efficacy. This technique, however, requires multiple forward passes through the network during inference and therefore can be too resource-intensive to be deployed in real-time applications. We propose a simple, easy-to-optimize distillation method for learning the conditional predictive distribution of a pre-trained dropout model for fast, sample-free uncertain…
Chalmers University of Technology（查尔姆斯理工大学）；Linköping University（林雪平大学）
Ensembles of neural networks have been shown to give better performance than single networks, both in terms of predictions and uncertainty estimation. Additionally, ensembles allow the uncertainty to be decomposed into aleatoric (data) and epistemic (model) components, giving a more complete picture of the predictive uncertainty. Ensemble distillation is the process of compressing an ensemble into a single model, often resulting in a leaner model that still outperforms the individual ensemble members. Unfortunately, standard distillation erases the natural uncertainty decomposition of the ense…
Cambridge University Engineering Department（剑桥大学工程系）；Yandex（Yandex公司）
Ensemble approaches are commonly used techniques to improving a system by combining multiple model predictions. Additionally these schemes allow the uncertainty, as well as the source of the uncertainty, to be derived for the prediction. Unfortunately these benefits come at a computational and memory cost. To address this problem ensemble distillation (EnD) and more recently ensemble distribution distillation (EnDD) have been proposed that compress the ensemble into a single model, representing either the ensemble average prediction or prediction distribution respectively. This paper examines …
Semantic Self-Distillation for Language Model Uncertainty
Department of Engineering Science University of Oxford（工程科学系牛津大学）；University of Oxford（牛津大学）；Oxford Suzhou Centre for Advanced Research University of Oxford Suzhou（牛津苏州先进研究中心）
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different \emph{forms} of uncertainty. However, ensembles come at a computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information abo…
Yandex（俄罗斯搜索引擎公司）；HSE University（高等经济大学）；University of Cambridge（剑桥大学）
Ensembles of machine learning models yield improved system performance as well as robust and interpretable uncertainty estimates; however, their inference costs may often be prohibitively high. \emph{Ensemble Distribution Distillation} is an approach that allows a single model to efficiently capture both the predictive performance and uncertainty estimates of an ensemble. For classification, this is achieved by training a Dirichlet distribution over the ensemble members' output distributions via the maximum likelihood criterion. Although theoretically principled, this criterion exhibits poor c…
Uncertainty Modeling for Multi-Objective RTA Interception with Distillation Acceleration
Department of Applied Mathematics（应用数学系）；Harbin Institute of Technology, Weihai（哈尔滨工业大学威海学院）；School of Mathematics and Statistics（数学与统计学学院）；Shandong University（山东大学）
University of Cambridge（剑桥大学）；University College London（伦敦大学学院）；University of Bremen（不来梅大学）；The Chinese University of Hong Kong（香港中文大学）
Existing deep-learning based tomographic image reconstruction methods do not provide accurate estimates of reconstruction uncertainty, hindering their real-world deployment. This paper develops a method, termed as the linearised deep image prior (DIP), to estimate the uncertainty associated with reconstructions produced by the DIP with total variation regularisation (TV). Specifically, we endow the DIP with conjugate Gaussian-linear model type error-bars computed from a local linearisation of the neural network around its optimised parameters. To preserve conjugacy, we approximate the TV regul…
