In recent years, there has been a great deal of research in developing end-to-end speech recognition models, which enable simplifying the traditional pipeline and achieving promising results. Despite their remarkable performance improvements, end-to-end models typically require expensive computational cost to show successful performance. To reduce this computational burden, knowledge distillation (KD), which is a popular model compression method, has been used to transfer knowledge from a deep and complex model (teacher) to a shallower and simpler model (student). Previous KD approaches have commonly designed the architecture of the student model by reducing the width per layer or the number of layers of the teacher model. This structural reduction scheme might limit the flexibility of model selection since the student model structure should be similar to that of the given teacher. To cope with this limitation, we propose a new KD method for end-to-end speech recognition, namely TutorNet, that can transfer knowledge across different types of neural networks at the hidden representation-level as well as the output-level. For concrete realizations, we firstly apply representation-level knowledge distillation (RKD) during the initialization step, and then apply the softmax-level knowledge distillation (SKD) combined with the original task learning. When the student is trained with RKD, we make use of frame weighting that points out the frames to which the teacher model pays more attention. Through a number of experiments on LibriSpeech dataset, it is verified that the proposed method not only distills the knowledge between networks with different topologies but also significantly contributes to improving the word error rate (WER) performance of the distilled student. Interestingly, TutorNet allows the student model to surpass its teacher's performance in some particular cases.
CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Models
Graduate Institute of Electrical Engineering, National Taiwan University（国立台湾大学电机工程学研究所）；Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通讯工程学研究所）；NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)（国立台湾大学人工智能卓越研究中心）
Fast Yet Effective Speech Emotion Recognition with Self-distillation
L3S Research Center, Leibniz University Hannover（汉诺威莱布尼茨大学L3S研究中心）；Griffith University（格里菲斯大学）；Imperial College London（帝国理工学院）；University of Augsburg（奥格斯堡大学）
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillation
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China（中国科学技术大学国家语言语音信息处理工程研究中心）
BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Alibaba Group（阿里巴巴集团）；Machine Intelligence Technology Lab, Alibaba Group（阿里巴巴集团机器智能技术实验室）
Multi-Level Knowledge Distillation for Speech Emotion Recognition in Noisy Conditions
Qingdao University of Science and Technology（青岛科技大学）；National Institute of Information and Communications Technology（国立信息通信研究机构）；Tianjin University（天津大学）
On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition
Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition
Leveraging Local and Global Knowledge Integration with Time-Frequency Calibrated Distillation for Speech Enhancement
School of Computer Science, Nanjing Audit University（南京审计大学计算机科学学院）；School of Communication Engineering, Nanjing Institute of Technology（南京工程技术学院通信工程学院）；School of Information Science and Engineering, Southeast University（东南大学信息科学与工程学院）；Cardiff University（卡迪夫大学）；Inner Mongolia University（内蒙古大学）；CHI – the Chair of Health Informatics, TUM University Hospital（健康信息学系，技术大学医院）；GLAM – the Group on Language, Audio, & Music, Imperial College London（语言、音频与音乐组，伦敦帝国理工学院）；Xiaomi EV（小米电动车）
Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
University of Michigan（密歇根大学）；National Taiwan University（国立台湾大学）；Signal Analysis and Interpretation Laboratory (SAIL)（信号分析与解释实验室）；University of Southern California（南加州大学）；National Taiwan University Artificial Intelligence Center of Research Excellence（国立台湾大学人工智能研究中心）
Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition
Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；MoE Key Lab of Artificial Intelligence, AI Institute（人工智能联合实验室）；X-LANCE Lab, Department of Computer Science and Engineering（X-LANCE实验室，计算机科学与工程系）；Shanghai Jiao Tong University（上海交通大学）；Peng Cheng Laboratory（鹏城实验室）
