This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers. Each head is trained in a self-supervised manner to mimic the main model's predictions, allowing computation to stop early when a calibrated confidence threshold is reached. We evaluate several confidence metrics and show that entropy provides the most reliable separation between correct and incorrect predictions. Experiments on the Pythia model suite (70M to 2.8B parameters) demonstrate that our method significantly reduces inference cost while maintaining accuracy across multiple benchmarks. We further adapt this approach to speculative decoding, introducing Dynamic Self-Speculative Decoding (DSSD), which achieves 1.66x higher token acceptance than manually-tuned LayerSkip baselines with minimal hyperparameter tuning.
University of Waterloo（滑铁卢大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）
Large-scale pre-trained language models such as BERT have brought significant improvements to NLP applications. However, they are also notorious for being slow in inference, which makes them difficult to deploy in real-time applications. We propose a simple but effective method, DeeBERT, to accelerate BERT inference. Our approach allows samples to exit earlier without passing through the entire model. Experiments show that DeeBERT is able to save up to ~40% inference time with minimal degradation in model quality. Further analyses show different behaviors in the BERT transformer layers and als…
Pre-training and then fine-tuning large language models is commonly used to achieve state-of-the-art performance in natural language processing (NLP) tasks. However, most pre-trained models suffer from low inference speed. Deploying such large models to applications with latency constraints is challenging. In this work, we focus on accelerating the inference via conditional computations. To achieve this, we propose a novel idea, Magic Pyramid (MP), to reduce both width-wise and depth-wise computation via token pruning and early exiting for Transformer-based models, particularly BERT. The forme…
Dynamic early exiting aims to accelerate the inference of pre-trained language models (PLMs) by emitting predictions in internal layers without passing through the entire model. In this paper, we empirically analyze the working mechanism of dynamic early exiting and find that it faces a performance bottleneck under high speed-up ratios. On one hand, the PLMs' representations in shallow layers lack high-level semantic information and thus are not sufficient for accurate predictions. On the other hand, the exiting decisions made by internal classifiers are unreliable, leading to wrongly emitted …
Despite the great success in Natural Language Processing (NLP) area, large pre-trained language models like BERT are not well-suited for resource-constrained or real-time applications owing to the large number of parameters and slow inference speed. Recently, compressing and accelerating BERT have become important topics. By incorporating a parameter-sharing strategy, ALBERT greatly reduces the number of parameters while achieving competitive performance. Nevertheless, ALBERT still suffers from a long inference time. In this work, we propose the ELBERT, which significantly improves the average…
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）
DE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks
School of Information Engineering, Tianjin University of Commerce（天津商业大学信息工程学院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Artificial Intelligence and Computer Science, Nantong University（南通大学人工智能与计算机科学学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；DataX Research Centre, School of Computing, Macquarie University（麦考瑞大学计算学院DataX研究中心）
Both performance and efficiency are crucial factors for sequence labeling tasks in many real-world scenarios. Although the pre-trained models (PTMs) have significantly improved the performance of various sequence labeling tasks, their computational cost is expensive. To alleviate this problem, we extend the recent successful early-exit mechanism to accelerate the inference of PTMs for sequence labeling tasks. However, existing early-exit mechanisms are specifically designed for sequence-level tasks, rather than sequence labeling. In this paper, we first propose a simple extension of sentence-l…
You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model
North Carolina State University（北卡罗来纳州立大学）；Google Research（谷歌研究院）；Northeastern University（东北大学）；University of Michigan（密歇根大学）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Connecticut（康涅狄格大学）
Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
School of information（信息学院）；Electronics, Beijing Institute of Technology, Beijing（电子学院，北京理工大学，北京）；Department of Computer Science, City University of Hong Kong, Hong Kong（计算机科学系，香港城市大学，香港）；School of Computer Science, Wuhan University, Wuhan（计算机科学学院，武汉大学，武汉）；School of Economics（经济学院）；Management, Communication University of China, Beijing（管理学院，中国传媒大学，北京）；School of Science（科学学院）；Engineering, The Chinese University of Hong Kong (Shenzhen), Shenzhen（工程学院，香港中文大学（深圳），深圳）
A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
Amirkabir University of Technology（阿米尔卡比尔理工大学）；Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）；Columbia University（哥伦比亚大学）；Amazon GenAI（亚马逊生成式AI部门）
Deep Neural Networks (DNNs) are generally designed as sequentially cascaded differentiable blocks/layers with a prediction module connected only to its last layer. DNNs can be attached with prediction modules at multiple points along the backbone where inference can stop at an intermediary stage without passing through all the modules. The last exit point may offer a better prediction error but also involves more computational resources and latency. An exit point that is `optimal' in terms of both prediction error and cost is desirable. The optimal exit point may depend on the latent distribut…
University of Shanghai for Science and Technology（上海理工大学）；State Street Corporation（道富公司）
There is an increasing need for more automated system-log analysis tools for large scale online system in a timely manner. However, conventional way to monitor and classify the log output based on keyword list does not scale well for complex system in which codes contributed by a large group of developers, with diverse ways of encoding the error messages, often with misleading pre-set labels. In this paper, we propose that the design of a large scale online log analysis should follow the "Least Prior Knowledge Principle", in which unsupervised or semi-supervised solution with the minimal prior…
Transformer-based self-supervised models are trained as feature extractors and have empowered many downstream speech tasks to achieve state-of-the-art performance. However, both the training and inference process of these models may encounter prohibitively high computational cost and large parameter budget. Although Parameter Sharing Strategy (PSS) proposed in ALBERT paves the way for parameter reduction, the computation required remains the same. Interestingly, we found in experiments that distributions of feature embeddings from different Transformer layers are similar when PSS is integrated…
Training dynamic models using early exits for automatic speech recognition on resource-constrained devices
University of Trento（特伦托大学）；LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎综合理工学院电信巴黎分校LTCI实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）；Universite Paris-Saclay, LISN, CNRS（巴黎萨克雷大学LISN实验室，法国国家科学研究中心）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
Institute of Digital Twin, Eastern Institute of Technology（数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；Munich Center for Machine Learning, LMU Munich（慕尼黑大学机器学习中心，慕尼黑大学）
In this paper, we address customer review understanding problems by using supervised machine learning approaches, in order to achieve a fully automatic review aspects categorisation and sentiment analysis. In general, such supervised learning algorithms require domain-specific expert knowledge for generating high quality labeled training data, and the cost of labeling can be very high. To achieve an in-production customer review machine learning enabled analysis tool with only a limited amount of data and within a reasonable training data collection time, we propose to use pre-trained language…
Self-supervised learning (SSL) models reshaped our approach to speech, language and vision. However their huge size and the opaque relations between their layers and tasks result in slow inference and network overthinking, where predictions made from the last layer of large models is worse than those made from intermediate layers. Early exit (EE) strategies can solve both issues by dynamically reducing computations at inference time for certain samples. Although popular for classification tasks in vision and language, EE has seen less use for sequence-to-sequence speech recognition (ASR) tasks…
Text analytics based on supervised machine learning classifiers has shown great promise in a multitude of domains, but has yet to be applied to Seismology. We test various standard models (Naive Bayes, k-Nearest Neighbors, Support Vector Machines, and Random Forests) on a seismological corpus of 100 articles related to the topic of precursory accelerating seismicity, spanning from 1988 to 2010. This corpus was labelled in Mignan (2011) with the precursor whether explained by critical processes (i.e., cascade triggering) or by other processes (such as signature of main fault loading). We invest…
Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfaction
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Scale AI Research Chair in Data-Driven Algorithms for Modern Supply Chains（数据驱动算法现代供应链研究席位）
In this paper, we introduce the Chinese corpus from CLUE organization, CLUECorpus2020, a large-scale corpus that can be used directly for self-supervised learning such as pre-training of a language model, or language generation. It has 100G raw corpus with 35 billion Chinese characters, which is retrieved from Common Crawl. To better understand this corpus, we conduct language understanding experiments on both small and large scale, and results show that the models trained on this corpus can achieve excellent performance on Chinese. We release a new Chinese vocabulary with a size of 8K, which …
