This paper is aimed at developing a method that reduces the computational cost of convolutional neural networks (CNN) during inference. Conventionally, the input data pass through a fixed neural network architecture. However, easy examples can be classified at early stages of processing and conventional networks do not take this into account. In this paper, we introduce 'Early-exit CNNs', EENets for short, which adapt their computational cost based on the input by stopping the inference process at certain exit locations. In EENets, there are a number of exit blocks each of which consists of a confidence branch and a softmax branch. The confidence branch computes the confidence score of exiting (i.e. stopping the inference process) at that location; while the softmax branch outputs a classification probability vector. Both branches are learnable and their parameters are separate. During training of EENets, in addition to the classical classification loss, the computational cost of inference is taken into account as well. As a result, the network adapts its many confidence branches to the inputs so that less computation is spent for easy examples. Inference works as in conventional feed-forward networks, however, when the output of a confidence branch is larger than a certain threshold, the inference stops for that specific example. The idea of EENets is applicable to available CNN architectures such as ResNets. Through comprehensive experiments on MNIST, SVHN, CIFAR10 and Tiny-ImageNet datasets, we show that early-exit (EE) ResNets achieve similar accuracy with their non-EE versions while reducing the computational cost to 20% of the original. Code is available at https://github.com/eksuas/eenets.pytorch
Today, artificial neural networks are the state of the art for solving a variety of complex tasks, especially in image classification. Such architectures consist of a sequence of stacked layers with the aim of extracting useful information and having it processed by a classifier to make accurate predictions. However, intermediate information within such models is often left unused. In other cases, such as in edge computing contexts, these architectures are divided into multiple partitions that are made functional by including early exits, i.e. intermediate classifiers, with the goal of reducin…
Confidence-gated training for efficient early-exit neural networks
International University of Rabat（拉巴特国际大学）；TICLab（TIC实验室）；Thales Alenia Space（泰勒斯阿莱尼亚空间）；Mohammed 6 Polytechnic University（穆莱·6技术大学）；University of Leeds（利兹大学）
Bridging Efficiency and Safety: Formal Verification of Neural Networks with Early Exits
The Hebrew University of Jerusalem, Israel（特拉维夫大学）；Bar Ilan University, Israel（巴伊兰大学）；Ben-Gurion University of the Negev, Israel（贝纳亚克大学）；University of California, Riverside, USA（加州大学河滨分校）
Early Exiting Predictive Coding Neural Networks for Edge AI
TICLab, International University of Rabat, Morocco（TICLab，拉巴特国际大学，摩洛哥）；College of Computing, University Mohammed VI Polytechnic, Morocco（计算机学院，穆罕默德六世理工大学，摩洛哥）；School of Electronic and Electrical Engineering, University of Leeds, UK（电子与电气工程学院，利兹大学，英国）
DistrEE: Distributed Early Exit of Deep Neural Network Inference on Edge Devices
School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications（北京邮电大学计算机学院（国家示范性软件学院））；School of Electronic Engineering, Beijing University of Posts and Telecommunications（北京邮电大学电子工程学院）
Temporal Decisions: Leveraging Temporal Correlation for Efficient Decisions in Early Exit Neural Networks
Infineon Technologies Dresden GmbH & Co. KG（英飞凌德累斯顿有限两合公司）；Technical University of Munich（慕尼黑工业大学）；Infineon Technologies AG（英飞凌股份公司）；Technical University of Dresden（德累斯顿工业大学）
Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization
Department of Computer Science and Engineering, IIT Bombay（印度理工学院班加罗尔计算机科学与工程系）；Center of Machine Intelligence and Data Science (C-MInDS), IIT Bombay（印度理工学院班加罗尔人工智能与数据科学中心）
E4: Energy-Efficient DNN Inference for Edge Video Analytics Via Early-Exit and DVFS
Harbin Institute of Technology（哈尔滨工业大学）；International Research Institute for Artificial Intelligence, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）国际人工智能研究院）；University at Albany - SUNY（纽约州立大学奥尔巴尼分校）
Tsinghua University（清华大学）；Carnegie Mellon University（卡内基梅隆大学）；China Mobile Research Institute（中国移动研究院）
Early exiting is an effective paradigm for improving the inference efficiency of deep networks. By constructing classifiers with varying resource demands (the exits), such networks allow easy samples to be output at early exits, removing the need for executing deeper layers. While existing works mainly focus on the architectural design of multi-exit networks, the training strategies for such models are largely left unexplored. The current state-of-the-art models treat all samples the same during training. However, the early-exiting behavior during testing has been ignored, leading to a gap bet…
Deploying deep learning services for time-sensitive and resource-constrained settings such as IoT using edge computing systems is a challenging task that requires dynamic adjustment of inference time. Multi-exit architectures allow deep neural networks to terminate their execution early in order to adhere to tight deadlines at the cost of accuracy. To mitigate this cost, in this paper we introduce a novel method called Multi-Exit Curriculum Learning that utilizes curriculum learning, a training strategy for neural networks that imitates human learning by sorting the training samples based on t…
Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well …
Multi-Exit models (MEMs) use an early-exit strategy to improve the accuracy and efficiency of deep neural networks (DNNs) by allowing samples to exit the network before the last layer. However, the effectiveness of MEMs in the presence of distribution shifts remains largely unexplored. Our work examines how distribution shifts generated by common image corruptions affect the accuracy/efficiency of MEMs. We find that under common corruptions, early-exiting at the first correct exit reduces the inference cost and provides a significant boost in accuracy ( 10%) over exiting at the last layer. How…
Deep learning models that perform well often have high computational costs. In this paper, we combine two approaches that try to reduce the computational cost while keeping the model performance high: pruning and early exit networks. We evaluate two approaches of pruning early exit networks: (1) pruning the entire network at once, (2) pruning the base network and additional linear classifiers in an ordered fashion. Experimental results show that pruning the entire network at once is a better strategy in general. However, at high accuracy rates, the two approaches have a similar performance, wh…
University of Southampton（南安普顿大学）；King’s College London（伦敦国王学院）
Transferring image-based object detectors to the domain of video remains challenging under resource constraints. Previous efforts utilised optical flow to allow unchanged features to be propagated, however, the overhead is considerable when working with very slowly changing scenes from applications such as surveillance. In this paper, we propose temporal early exits to reduce the computational complexity of per-frame video object detection. Multiple temporal early exit modules with low computational overhead are inserted at early layers of the backbone network to identify the semantic differen…
We propose Class Based Thresholding (CBT) to reduce the computational cost of early exit semantic segmentation models while preserving the mean intersection over union (mIoU) performance. A key idea of CBT is to exploit the naturally-occurring neural collapse phenomenon. Specifically, by calculating the mean prediction probabilities of each class in the training set, CBT assigns different masking threshold values to each class, so that the computation can be terminated sooner for pixels belonging to easy-to-predict classes. We show the effectiveness of CBT on Cityscapes and ADE20K datasets. CB…
Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks
TICLab, International University of Rabat, Morocco（拉巴特国际大学TIC实验室，摩洛哥）；College of Computing, University Mohammed VI Polytechnic, Morocco（摩洛哥穆莱·伊斯梅尔Polytechnic大学计算机学院）；School of Electronic and Electrical Engineering, University of Leeds, UK（利兹大学电子与电气工程学院）
ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation
Fraunhofer Institute for Computer Graphics Research IGD, Germany（德国弗劳恩霍夫计算机图形学研究所IGD）；Department of Computer Science, TU Darmstadt, Germany（德国达姆施塔特工业大学计算机科学系）
You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model
North Carolina State University（北卡罗来纳州立大学）；Google Research（谷歌研究院）；Northeastern University（东北大学）；University of Michigan（密歇根大学）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Connecticut（康涅狄格大学）
In this paper, we propose a conditional early exiting framework for efficient video recognition. While existing works focus on selecting a subset of salient frames to reduce the computation costs, we propose to use a simple sampling strategy combined with conditional early exiting to enable efficient recognition. Our model automatically learns to process fewer frames for simpler videos and more frames for complex ones. To achieve this, we employ a cascade of gating modules to automatically determine the earliest point in processing where an inference is sufficiently reliable. We generate on-th…
CalexNet: Soft Cascade-Aligned Training and Calibration for Lightweight Early-Exit Branches
Intelligent Systems, Afeka Academic College of Engineering（阿法卡学术工程学院智能系统）；School of Computer Science, Faculty of Sciences, Holon Institute of Technology（霍洛恩理工学院计算机科学学院）
Samsung AI Center, Cambridge（三星人工智能中心（剑桥））；University of Cambridge（剑桥大学）
Convolutional neural networks (CNNs) have recently become the state-of-the-art in a diversity of AI tasks. Despite their popularity, CNN inference still comes at a high computational cost. A growing body of work aims to alleviate this by exploiting the difference in the classification difficulty among samples and early-exiting at different stages of the network. Nevertheless, existing studies on early exiting have primarily focused on the training scheme, without considering the use-case requirements or the deployment platform. This work presents HAPI, a novel methodology for generating high-p…
This paper proposes a novel knowledge distillation-based learning method to improve the classification performance of convolutional neural networks (CNNs) without a pre-trained teacher network, called exit-ensemble distillation. Our method exploits the multi-exit architecture that adds auxiliary classifiers (called exits) in the middle of a conventional CNN, through which early inference results can be obtained. The idea of our method is to train the network using the ensemble of the exits as the distillation target, which greatly improves the classification performance of the overall network.…
The use of Neural Architecture Search (NAS) techniques to automate the design of neural networks has become increasingly popular in recent years. The proliferation of devices with different hardware characteristics using such neural networks, as well as the need to reduce the power consumption for their search, has led to the realisation of Once-For-All (OFA), an eco-friendly algorithm characterised by the ability to generate easily adaptable models through a single learning process. In order to improve this paradigm and develop high-performance yet eco-friendly NAS techniques, this paper pres…
Deep neural networks can be converted to multi-exit architectures by inserting early exit branches after some of their intermediate layers. This allows their inference process to become dynamic, which is useful for time critical IoT applications with stringent latency requirements, but with time-variant communication and computation resources. In particular, in edge computing systems and IoT networks where the exact computation time budget is variable and not known beforehand. Vision Transformer is a recently proposed architecture which has since found many applications across various domains …
Samsung AI Center（三星人工智能中心）；University of Cambridge（剑桥大学）
On-device machine learning is becoming a reality thanks to the availability of powerful hardware and model compression techniques. Typically, these models are pretrained on large GPU clusters and have enough parameters to generalise across a wide variety of inputs. In this work, we observe that a much smaller, personalised model can be employed to fit a specific scenario, resulting in both higher accuracy and faster execution. Nevertheless, on-device training is extremely challenging, imposing excessive computational and memory requirements even for flagship smartphones. At the same time, on-d…
The natural world often follows a long-tailed data distribution where only a few classes account for most of the examples. This long-tail causes classifiers to overfit to the majority class. To mitigate this, prior solutions commonly adopt class rebalancing strategies such as data resampling and loss reshaping. However, by treating each example within a class equally, these methods fail to account for the important notion of example hardness, i.e., within each class some examples are easier to classify than others. To incorporate this notion of hardness into the learning process, we propose th…
Deep neural networks are state of the art methods for many learning tasks due to their ability to extract increasingly better features at each network layer. However, the improved performance of additional layers in a deep network comes at the cost of added latency and energy usage in feedforward inference. As networks continue to get deeper and larger, these costs become more prohibitive for real-time and energy-sensitive applications. To address this issue, we present BranchyNet, a novel deep network architecture that is augmented with additional side branch classifiers. The architecture all…
