Speech emotion recognition (SER) is an important research topic in human-computer interaction. Existing works mainly rely on human expertise to design models. Despite their success, different datasets often require distinct structures and hyperparameters. Searching for an optimal model for each dataset is time-consuming and labor-intensive. To address this problem, we propose a two-stream neural architecture search (NAS) based framework, called \enquote{EmotionNAS}. Specifically, we take two-stream features (i.e., handcrafted and deep features) as the inputs, followed by NAS to search for the optimal structure for each stream. Furthermore, we incorporate complementary information in different streams through an efficient information supplement module. Experimental results demonstrate that our method outperforms existing manually-designed and NAS-based models, setting the new state-of-the-art record.
University of Augsburg（奥格斯堡大学）；SyncPilot GmbH（SyncPilot有限公司）；Imperial College London（帝国理工学院）
Packet loss is a common problem in data transmission, including speech data transmission. This may affect a wide range of applications that stream audio data, like streaming applications or speech emotion recognition (SER). Packet Loss Concealment (PLC) is any technique of facing packet loss. Simple PLC baselines are 0-substitution or linear interpolation. In this paper, we present a concealment wrapper, which can be used with stacked recurrent neural cells. The concealment cell can provide a recurrent neural network (ConcealNet), that performs real-time step-wise end-to-end PLC at inference t…
University of Augsburg（奥格斯堡大学）；SyncPilot GmbH（SyncPilot有限公司）；Imperial College London（帝国理工学院）
In applications that use emotion recognition via speech, frame-loss can be a severe issue given manifold applications, where the audio stream loses some data frames, for a variety of reasons like low bandwidth. In this contribution, we investigate for the first time the effects of frame-loss on the performance of emotion recognition via speech. Reproducible extensive experiments are reported on the popular RECOLA corpus using a state-of-the-art end-to-end deep neural network, which mainly consists of convolution blocks and recurrent layers. A simple environment based on a Markov Chain model is…
Speech emotion recognition systems have high prediction latency because of the high computational requirements for deep learning models and low generalizability mainly because of the poor reliability of emotional measurements across multiple corpora. To solve these problems, we present a speech emotion recognition system based on a reductionist approach of decomposing and analyzing syllable-level features. Mel-spectrogram of an audio stream is decomposed into syllable-level components, which are then analyzed to extract statistical features. The proposed method uses formant attention, noise-ga…
University of Science and Technology of China（中国科学技术大学）；Georgia Institute of Technology（佐治亚理工学院）
Multimodal emotion recognition is a challenging task in emotion computing as it is quite difficult to extract discriminative features to identify the subtle differences in human emotions with abstract concept and multiple expressions. Moreover, how to fully utilize both audio and visual information is still an open problem. In this paper, we propose a novel multimodal fusion attention network for audio-visual emotion recognition based on adaptive and multi-level factorized bilinear pooling (FBP). First, for the audio stream, a fully convolutional network (FCN) equipped with 1-D attention mecha…
University of Augsburg（奥格斯堡大学）；Huawei Technologies（华为技术有限公司）；HSE University（俄罗斯国家研究型高等经济大学）；ITMO University（圣彼得堡国立信息技术机械与光学大学）；Imperial College London（伦敦帝国理工学院）
Text encodings from automatic speech recognition (ASR) transcripts and audio representations have shown promise in speech emotion recognition (SER) ever since. Yet, it is challenging to explain the effect of each information stream on the SER systems. Further, more clarification is required for analysing the impact of ASR's word error rate (WER) on linguistic emotion recognition per se and in the context of fusion with acoustic information exploitation in the age of deep ASR systems. In order to tackle the above issues, we create transcripts from the original speech by applying three modern AS…
Emotion recognition in conversation (ERC) has attracted much attention in recent years for its necessity in widespread applications. Existing ERC methods mostly model the self and inter-speaker context separately, posing a major issue for lacking enough interaction between them. In this paper, we propose a novel Speaker and Position-Aware Graph neural network model for ERC (S+PAGE), which contains three stages to combine the benefits of both Transformer and relational graph convolution network (R-GCN) for better contextual modeling. Firstly, a two-stream conversational Transformer is presented…
Audio-Video Emotion Recognition is now attacked with Deep Neural Network modeling tools. In published papers, as a rule, the authors show only cases of the superiority in multi-modality over audio-only or video-only modality. However, there are cases superiority in uni-modality can be found. In our research, we hypothesize that for fuzzy categories of emotional events, the within-modal and inter-modal noisy information represented indirectly in the parameters of the modeling neural network impedes better performance in the existing late fusion and end-to-end multi-modal network training strate…
LLM-Based Synthetic Ground Truth Generation for Audio-Based Emotion Classification via In-Context Learning
School of Business, Technical University of Applied Sciences Augsburg（应用技术大学阿沙芬堡商学院）；Data Science und Autonome Systeme Technologietransferzentrum (TTZ)（数据科学与自主系统技术转移中心（TTZ））
Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition
College of Computer and Mathematics, Central South University of Forestry and Technology（中南林业科技大学计算机与数学学院）；Department of Computer Science, State University of New York, New Paltz（纽约州立大学新帕尔茨分校计算机科学系）
Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models
Nanyang Technological University, Singapore（南洋理工大学，新加坡）；Institute for Infocomm Research, A STAR, Singapore（信息通信研究所，新加坡）；The Hong Kong Polytechnic University, Hong Kong（香港理工大学，香港）
NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation（日本电信电话株式会社 NTT 通信科学研究所）
This paper proposes a non-autoregressive extension of our previously proposed sequence-to-sequence (S2S) model-based voice conversion (VC) methods. S2S model-based VC methods have attracted particular attention in recent years for their flexibility in converting not only the voice identity but also the pitch contour and local duration of input speech, thanks to the ability of the encoder-decoder architecture with the attention mechanism. However, one of the obstacles to making these methods work in real-time is the autoregressive (AR) structure. To overcome this obstacle, we develop a method t…
SuZhou Automotive Research Institute of Tsinghua University（清华大学苏州汽车研究院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Hyundai Motor Advanced Tech. R&D Center；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）
Unified Architecture and Unsupervised Speech Disentanglement for Speaker Embedding-Free Enrollment in Personalized Speech Enhancement
Shanghai Engineering Research Center of Intelligent Education and Bigdata, Shanghai Normal University（智能教育与大数据上海工程研究中心，上海师范大学）；SHNU-Unisound Natural Human-Computer Interaction Lab, Shanghai Normal University（上海师范大学Unisound人机交互实验室）；Unisound AI Technology Co., Ltd.（Unisound人工智能技术有限公司）
OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究）；GWM AI Lab（GWM人工智能实验室）
