Estimating dimensional emotions, such as activation, valence and dominance, from acoustic speech signals has been widely explored over the past few years. While accurate estimation of activation and dominance from speech seem to be possible, the same for valence remains challenging. Previous research has shown that the use of lexical information can improve valence estimation performance. Lexical information can be obtained from pre-trained acoustic models, where the learned representations can improve valence estimation from speech. We investigate the use of pre-trained model representations to improve valence estimation from acoustic speech signal. We also explore fusion of representations to improve emotion estimation across all three emotion dimensions: activation, valence and dominance. Additionally, we investigate if representations from pre-trained models can be distilled into models trained with low-level features, resulting in models with a less number of parameters. We show that fusion of pre-trained model embeddings result in a 79% relative improvement in concordance correlation coefficient CCC on valence estimation compared to standard acoustic feature baseline (mel-filterbank energies), while distillation from pre-trained model embeddings to lower-dimensional representations yielded a relative 12% improvement. Such performance gains were observed over two evaluation sets, indicating that our proposed architecture generalizes across those evaluation sets. We report new state-of-the-art "text-free" acoustic-only dimensional emotion estimation $CCC$ values on two MSP-Podcast evaluation sets.
Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition
Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation
School of Computer Science and Technology, China University of Mining and Technology（计算机科学与技术学院，中国矿业大学）；Mine Digitization Engineering Research Center of Ministry of Education, China University of Mining and Technology（教育部矿山数字化工程研究中心，中国矿业大学）
Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification
NLP Group, Pompeu Fabra University, Barcelona, Spain（自然语言处理组，庞培法布拉大学，巴塞罗那，西班牙）；Language Technologies Unit, Barcelona Supercomputing Center, Spain（语言技术单元，巴塞罗那超级计算中心，西班牙）
Investigating the Robustness of Subtask Distillation under Spurious Correlation
1Machine Learning Group, Technische Universit\" a t Berlin (TU Berlin), 10587 Berlin, Germany 2Konrad Zuse School of Excellence in Learning；Intelligent Systems (ELIZA), 64289 Darmstadt, Germany 3BIFOLD\,--\,Berlin Institute for the Foundations of Learning；Data, 10587 Berlin, Germany 4Department of Artificial Intelligence, Korea University, Seoul 136-713, Korea 5Max Planck Institute for Informatics, 66123 Saarbr \"u cken, Germany 6Institute for AI in Medicine, Charit\'e\,--\,Universit\"atsmedizin Berlin, 10115 Berlin, Germany
Speech Emotion Recognition (SER) is one of the essential perceptual methods of humans in understanding the situation and how to interact with others, therefore, in recent years, it has been tried to add the ability to recognize emotions to human-machine communication systems. Since the SER process relies on labeled data, databases are essential for it. Incomplete, low-quality or defective data may lead to inaccurate predictions. In this paper, we fixed the inconsistencies in Sharif Emotional Speech Database (ShEMO), as a Persian database, by using an Automatic Speech Recognition (ASR) system a…
Pre-trained model representations have demonstrated state-of-the-art performance in speech recognition, natural language processing, and other applications. Speech models, such as Bidirectional Encoder Representations from Transformers (BERT) and Hidden units BERT (HuBERT), have enabled generating lexical and acoustic representations to benefit speech recognition applications. We investigated the use of pre-trained model representations for estimating dimensional emotions, such as activation, valence, and dominance, from speech. We observed that while valence may rely heavily on lexical repres…
Powder AI Research（Powder AI 研究院）；Université Paris-Est, LISSI, UPEC（巴黎东大学 LISSI 实验室 / 巴黎东克雷泰伊大学）
Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation …
National Taiwan University（台湾大学）；ASUS Intelligent Cloud Services（华硕智能云服务）
Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-supervised speech models. In our work, we performed Ensemble Knowledge Distillation (EKD) on various self-supervised speech models such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective. On top of that, we proposed a multiple prediction head method …
Breaking Resource Barriers in Speech Emotion Recognition via Data Distillation
Imperial College London（伦敦帝国学院）；University of Bremen（不莱梅大学）；Beijing Institute of Technology（北京理工大学）；Griffith University（格里菲斯大学）；Technical University of Munich (TUM)（慕尼黑技术大学）；GLAM – the Group on Language, Audio, & Music Cognitive Systems Lab（语言、音频与音乐认知系统实验室）；School of Medical Technology（医学技术学院）；Klinikum rechts der Isar（莱茵河右岸医院）
Tianjin Normal University（天津师范大学）；Zhejiang University（浙江大学）；University of Augsburg（奥格斯堡大学）；Imperial College London（帝国理工学院）
The goal of Speech Emotion Recognition (SER) is to enable computers to recognize the emotion category of a given utterance in the same way that humans do. The accuracy of SER is strongly dependent on the validity of the utterance-level representation obtained by the model. Nevertheless, the ``dark knowledge" carried by non-target classes is always ignored by previous studies. In this paper, we propose a hierarchical network, called DKDFMH, which employs decoupled knowledge distillation in a deep convolutional neural network with a fused multi-head attention mechanism. Our approach applies logi…
L3S Research Center, Leibniz University Hannover（汉诺威莱布尼茨大学L3S研究中心）；Griffith University（格里菲斯大学）；Imperial College London（帝国理工学院）；University of Augsburg（奥格斯堡大学）
Speech emotion recognition (SER) is the task of recognising human's emotional states from speech. SER is extremely prevalent in helping dialogue systems to truly understand our emotions and become a trustworthy human conversational partner. Due to the lengthy nature of speech, SER also suffers from the lack of abundant labelled data for powerful models like deep neural networks. Pre-trained complex models on large-scale speech datasets have been successfully applied to SER via transfer learning. However, fine-tuning complex models still requires large memory space and results in low inference …
StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation
Nanyang Technological University, Singapore（南洋理工大学）；Institute for Infocomm Research (I2R), A*STAR, Singapore（信息与通信研究所）；The Hong Kong Polytechnic University, Hong Kong（香港理工大学）
Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time. In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of t…
Investigating Acoustic-Textual Emotional Inconsistency Information for Automatic Depression Detection
Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院粤港澳人机协同智能系统联合实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Good Mood Health Industry Group Co., Ltd（好心情健康产业集团有限公司）；Shanghai Jiao Tong University（上海交通大学）
Multi-Level Knowledge Distillation for Speech Emotion Recognition in Noisy Conditions
Qingdao University of Science and Technology（青岛科技大学）；National Institute of Information and Communications Technology（国立信息通信研究机构）；Tianjin University（天津大学）
Sharif University of Technology（谢里夫理工大学）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
The increased amount of multi-modal medical data has opened the opportunities to simultaneously process various modalities such as imaging and non-imaging data to gain a comprehensive insight into the disease prediction domain. Recent studies using Graph Convolutional Networks (GCNs) provide novel semi-supervised approaches for integrating heterogeneous modalities while investigating the patients' associations for disease prediction. However, when the meta-data used for graph construction is not available at inference time (e.g., coming from a distinct population), the conventional methods exh…
Investigating Polyglot Speech Foundation Models for Learning Collective Emotion from Crowds
IIIT-Delhi（印度德里印度理工学院）；UPES（乌普萨拉理工学院）；V.B.S.P.U（V.B.S.P.U大学）；VIT（瓦拉纳西理工学院）；Independent Researcher（独立研究者）；L.B.R College of Engineering（L.B.R工程学院）；Reliance AI
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
This paper investigates a novel task of talking face video generation solely from speeches. The speech-to-video generation technique can spark interesting applications in entertainment, customer service, and human-computer-interaction industries. Indeed, the timbre, accent and speed in speeches could contain rich information relevant to speakers' appearance. The challenge mainly lies in disentangling the distinct visual attributes from audio signals. In this article, we propose a light-weight, cross-modal distillation method to extract disentangled emotional and identity information from unlab…
Human emotional speech is, by its very nature, a variant signal. This results in dynamics intrinsic to automatic emotion classification based on speech. In this work, we explore a spectral decomposition method stemming from fluid-dynamics, known as Dynamic Mode Decomposition (DMD), to computationally represent and analyze the global utterance-level dynamics of emotional speech. Specifically, segment-level emotion-specific representations are first learned through an Emotion Distillation process. This forms a multi-dimensional signal of emotion flow for each utterance, called Emotion Profiles (…
EmoTale: An Enacted Speech-emotion Dataset in Danish
University Grenoble Alpes, CNRS, Grenoble INP, LIG（格陵兰大学、CNRS、格陵兰INP、LIG）；Dept. of Applied Mathematics and Computer Science, Technical University of Denmark（应用数学和计算机科学系，丹麦技术大学）
Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition
MOE-Key Laboratory of Pervasive Computing, Department of Computer Science and Technology, Tsinghua University（MOE-Key pervasive computing laboratory，计算机科学与技术系，清华大学）；Key State Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播关键实验室，中国传媒大学）；Department of Computer Science and Computer Engineering, Texas A&M University（计算机科学与工程系，德克萨斯大学）
Emotions Beyond Words: Non-Speech Audio Emotion Recognition With Edge Computing
EmulationAI；Queensland University of Technology (QUT)（昆士兰科技大学（QUT））；University of the West Scotland（西苏格兰大学）；National University of Computer and Emerging Sciences (NUCES)（国立计算机与新兴科学大学（NUCES））；Qatar University（卡塔尔大学）
