The emergence of Large Audio-Language Models (LALMs) has advanced Speech Emotion Recognition (SER), but their size limits deployment in resource-constrained environments. While Knowledge Distillation is effective for LALM compression, existing methods remain underexplored in distilling the cross-modal projection module (Projector), and often struggle with alignment due to differences in feature dimensions. We propose PL-Distill, a KD framework that combines Projector-Level Distillation (PDist) to align audio embeddings and Logits-Level Distillation (LDist) to align output logits. PDist introduces Attention-weighted Centered Kernel Alignment, a novel approach we propose to highlight important time steps and address dimension mismatches. Meanwhile, LDist minimizes the Kullback-Leibler divergence between teacher and student logits from audio and text modalities. On IEMOCAP, RAVDESS, and SAVEE, PL-Distill compresses an 8.4B-parameter teacher to a compact 1.1B-parameter student, consistently outperforming the teacher, state-of-the-art pretrained models, and other KD baselines across all metrics.
Teaching Audio Models to Reason: A Unified Framework for Source- and Layer-wise Distillation
Jiutian Artificial Intelligence Research Institute, China Mobile, Beijing, China（九天人工智能研究院，中国移动，北京）；The State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China（多媒体信息处理国家重点实验室，北京大学，北京）
Tianjin Normal University（天津师范大学）；Zhejiang University（浙江大学）；University of Augsburg（奥格斯堡大学）；Imperial College London（帝国理工学院）
The goal of Speech Emotion Recognition (SER) is to enable computers to recognize the emotion category of a given utterance in the same way that humans do. The accuracy of SER is strongly dependent on the validity of the utterance-level representation obtained by the model. Nevertheless, the ``dark knowledge" carried by non-target classes is always ignored by previous studies. In this paper, we propose a hierarchical network, called DKDFMH, which employs decoupled knowledge distillation in a deep convolutional neural network with a fused multi-head attention mechanism. Our approach applies logi…
L3S Research Center, Leibniz University Hannover（汉诺威莱布尼茨大学L3S研究中心）；Griffith University（格里菲斯大学）；Imperial College London（帝国理工学院）；University of Augsburg（奥格斯堡大学）
Speech emotion recognition (SER) is the task of recognising human's emotional states from speech. SER is extremely prevalent in helping dialogue systems to truly understand our emotions and become a trustworthy human conversational partner. Due to the lengthy nature of speech, SER also suffers from the lack of abundant labelled data for powerful models like deep neural networks. Pre-trained complex models on large-scale speech datasets have been successfully applied to SER via transfer learning. However, fine-tuning complex models still requires large memory space and results in low inference …
Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition
Multi-Level Knowledge Distillation for Speech Emotion Recognition in Noisy Conditions
Qingdao University of Science and Technology（青岛科技大学）；National Institute of Information and Communications Technology（国立信息通信研究机构）；Tianjin University（天津大学）
Breaking Resource Barriers in Speech Emotion Recognition via Data Distillation
Imperial College London（伦敦帝国学院）；University of Bremen（不莱梅大学）；Beijing Institute of Technology（北京理工大学）；Griffith University（格里菲斯大学）；Technical University of Munich (TUM)（慕尼黑技术大学）；GLAM – the Group on Language, Audio, & Music Cognitive Systems Lab（语言、音频与音乐认知系统实验室）；School of Medical Technology（医学技术学院）；Klinikum rechts der Isar（莱茵河右岸医院）
Estimating dimensional emotions, such as activation, valence and dominance, from acoustic speech signals has been widely explored over the past few years. While accurate estimation of activation and dominance from speech seem to be possible, the same for valence remains challenging. Previous research has shown that the use of lexical information can improve valence estimation performance. Lexical information can be obtained from pre-trained acoustic models, where the learned representations can improve valence estimation from speech. We investigate the use of pre-trained model representations …
Automatic Analysis of the Emotional Content of Speech in Daylong Child-Centered Recordings from a Neonatal Intensive Care Unit
Unit of Computing Sciences, Tampere University, Finland（图瓦大学计算科学系）；Department of Clinical Medicine, University of Turku, Finland（图尔库大学临床医学系）；Department of Signal Processing and Acoustics, Aalto University, Finland（阿尔托大学信号处理与声学系）
Human emotional speech is, by its very nature, a variant signal. This results in dynamics intrinsic to automatic emotion classification based on speech. In this work, we explore a spectral decomposition method stemming from fluid-dynamics, known as Dynamic Mode Decomposition (DMD), to computationally represent and analyze the global utterance-level dynamics of emotional speech. Specifically, segment-level emotion-specific representations are first learned through an Emotion Distillation process. This forms a multi-dimensional signal of emotion flow for each utterance, called Emotion Profiles (…
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
This paper investigates a novel task of talking face video generation solely from speeches. The speech-to-video generation technique can spark interesting applications in entertainment, customer service, and human-computer-interaction industries. Indeed, the timbre, accent and speed in speeches could contain rich information relevant to speakers' appearance. The challenge mainly lies in disentangling the distinct visual attributes from audio signals. In this article, we propose a light-weight, cross-modal distillation method to extract disentangled emotional and identity information from unlab…
Emotions Beyond Words: Non-Speech Audio Emotion Recognition With Edge Computing
EmulationAI；Queensland University of Technology (QUT)（昆士兰科技大学（QUT））；University of the West Scotland（西苏格兰大学）；National University of Computer and Emerging Sciences (NUCES)（国立计算机与新兴科学大学（NUCES））；Qatar University（卡塔尔大学）
Nankai University（南开大学）；TMCC, College of Computer Science（TMCC计算机学院）；Hithink RoyalFlush AI Research Institute（Hithink RoyalFlush人工智能研究院）；University of Exeter（埃克塞特大学）
This work presents an initial proof of concept of how Music Emotion Recognition (MER) systems could be intentionally biased with respect to annotations of musically induced emotions in a political context. In specific, we analyze traditional Colombian music containing politically charged lyrics of two types: (1) vallenatos and social songs from the "left-wing" guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC) and (2) corridos from the "right-wing" paramilitaries Autodefensas Unidas de Colombia (AUC). We train personalized machine learning models to predict induced emotions for three…
Powder AI Research（Powder AI 研究院）；Université Paris-Est, LISSI, UPEC（巴黎东大学 LISSI 实验室 / 巴黎东克雷泰伊大学）
Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation …
National Taiwan University（台湾大学）；ASUS Intelligent Cloud Services（华硕智能云服务）
Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-supervised speech models. In our work, we performed Ensemble Knowledge Distillation (EKD) on various self-supervised speech models such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective. On top of that, we proposed a multiple prediction head method …
Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition
MOE-Key Laboratory of Pervasive Computing, Department of Computer Science and Technology, Tsinghua University（MOE-Key pervasive computing laboratory，计算机科学与技术系，清华大学）；Key State Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播关键实验室，中国传媒大学）；Department of Computer Science and Computer Engineering, Texas A&M University（计算机科学与工程系，德克萨斯大学）
Department of Mathematics and Computer Science, Eindhoven University of Technology（埃因霍温理工大学数学与计算机科学系）
Speech Emotion Recognition (SER) refers to the recognition of human emotions from natural speech. If done accurately, it can offer a number of benefits in building human-centered context-aware intelligent systems. Existing SER approaches are largely centralized, without considering users' privacy. Federated Learning (FL) is a distributed machine learning paradigm dealing with decentralization of privacy-sensitive personal data. In this paper, we present a privacy-preserving and data-efficient SER approach by utilizing the concept of FL. To the best of our knowledge, this is the first federated…
Pre-trained model representations have demonstrated state-of-the-art performance in speech recognition, natural language processing, and other applications. Speech models, such as Bidirectional Encoder Representations from Transformers (BERT) and Hidden units BERT (HuBERT), have enabled generating lexical and acoustic representations to benefit speech recognition applications. We investigated the use of pre-trained model representations for estimating dimensional emotions, such as activation, valence, and dominance, from speech. We observed that while valence may rely heavily on lexical repres…
VoiceWeaver: Staged Learning of Structured Controls for Expressive Speech and Sound-Event Generation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（上海人工智能教育研究院，华东师范大学，中国）；School of Computer Science and Technology, East China Normal University, China（计算机科学与技术学院，华东师范大学，中国）；Boin Hearing Technology (Shanghai) Co., LTD, China（博英听力技术（上海）有限公司，中国）；Department of Computing Science, University of Aberdeen, UK（计算科学系，阿伯丁大学，英国）
