State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compared to their Attention-based counterparts. On the other hand, the community has built up a considerable body of knowledge on how to train Transformers, and many pretrained Transformer models are readily available. To facilitate the adoption of SSMs while leveraging existing pretrained Transformers, we aim to identify an effective recipe to distill an Attention-based model into a Mamba-like architecture. In prior work on cross-architecture distillation, however, it has been shown that a naïve distillation procedure from Transformers to Mamba fails to preserve the original teacher performance, a limitation often overcome with hybrid solutions combining Attention and SSM blocks. The key argument from our work is that, by equipping Mamba with a principled initialization, we can recover an overall better recipe for cross-architectural distillation. To this end, we propose a principled two-stage approach: first, we distill knowledge from a traditional Transformer into a linearized version of Attention, using an adaptation of the kernel trick. Then, we distill the linearized version into an adapted Mamba model that does not use any Attention block. Overall, the distilled Mamba model is able to preserve the original Pythia-1B Transformer performance in downstream tasks, maintaining a perplexity of 14.11 close to the teacher's 13.86. To show the efficacy of our recipe, we conduct thorough ablations at 1B scale with 10B tokens varying sequence mixer architecture, scaling analysis on model sizes and total distillation tokens, and a sensitivity analysis on tokens allocation between stages.
In recent years, Natural Language Processing (NLP) models have achieved phenomenal success in linguistic and semantic tasks like text classification, machine translation, cognitive dialogue systems, information retrieval via Natural Language Understanding (NLU), and Natural Language Generation (NLG). This feat is primarily attributed due to the seminal Transformer architecture, leading to designs such as BERT, GPT (I, II, III), etc. Although these large-size models have achieved unprecedented performances, they come at high computational costs. Consequently, some of the recent NLP architecture…
Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation
SIA-Lab of Tsinghua AIR and ByteDance Seed（清华大学AIR与字节跳动种子SIA实验室）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中科院自动化所）；School of Artificial Intelligence, UCAS（中国科学技术大学人工智能学院）；Fudan University（复旦大学）；Independent Researcher（独立研究者）
Small Vision-Language Models: A Survey on Compact Architectures and Techniques
School of Computer Engineering, Kalinga Institute of Industrial Technology (KIIT) Deemed to be University（卡林加工业技术学院（KIIT）德emed大学计算机工程学院）；Indian Institute of Management Visakhapatnam（印度管理学院维沙卡帕特南分院）；Remodel AI
LastOPD: Taming Collapse in Latent On-Policy Distillation
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Visa Research（Visa研究院）；Case Western Reserve University（凯斯西储大学）；University of Florida（佛罗里达大学）；The Ohio State University（俄亥俄州立大学）
Transformer-based models are the state-of-the-art for Natural Language Understanding (NLU) applications. Models are getting bigger and better on various tasks. However, Transformer models remain computationally challenging since they are not efficient at inference-time compared to traditional approaches. In this paper, we present FastFormers, a set of recipes to achieve efficient inference-time performance for Transformer-based models on various NLU tasks. We show how carefully utilizing knowledge distillation, structured pruning and numerical optimization can lead to drastic improvements on i…
