The majority of existing speech emotion recognition models are trained and evaluated on a single corpus and a single language setting. These systems do not perform as well when applied in a cross-corpus and cross-language scenario. This paper presents results for speech emotion recognition for 4 languages in both single corpus and cross corpus setting. Additionally, since multi-task learning (MTL) with gender, naturalness and arousal as auxiliary tasks has shown to enhance the generalisation capabilities of the emotion models, this paper introduces language ID as another auxiliary task in MTL framework to explore the role of spoken language on emotion recognition which has not been studied yet.
The Experimental High School Attached to Beijing Normal University（北京师范大学附属实验中学）
A speech emotion recognition algorithm based on multi-feature and Multi-lingual fusion is proposed in order to resolve low recognition accuracy caused by lack of large speech dataset and low robustness of acoustic features in the recognition of speech emotion. First, handcrafted and deep automatic features are extracted from existing data in Chinese and English speech emotions. Then, the various features are fused respectively. Finally, the fused features of different languages are fused again and trained in a classification model. Distinguishing the fused features with the unfused ones, the r…
Semantic-Emotional Resonance Embedding: A Semi-Supervised Paradigm for Cross-Lingual Speech Emotion Recognition
School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；Pengcheng Laboratory Xinjiang Network Node（鹏城实验室新疆网络节点）；Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center（新疆多模态智能处理与信息安全工程技术研究中心）；Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学具身智能联合研究实验室）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语言认知计算国际联合研究实验室）
Expert System；Universitat Pompeu Fabra（庞培法布拉大学）；University of Oslo（奥斯陆大学）
Emotion intensity prediction determines the degree or intensity of an emotion that the author expresses in a text, extending previous categorical approaches to emotion detection. While most previous work on this topic has concentrated on English texts, other languages would also benefit from fine-grained emotion classification, preferably without having to recreate the amount of annotated data available in English in each new language. Consequently, we explore cross-lingual transfer approaches for fine-grained emotion detection in Spanish and Catalan tweets. To this end we annotate a test set …
Research on multilingual speech emotion recognition faces the problem that most available speech corpora differ from each other in important ways, such as annotation methods or interaction scenarios. These inconsistencies complicate building a multilingual system. We present results for cross-lingual and multilingual emotion recognition on English and French speech data with similar characteristics in terms of interaction (human-human conversations). Further, we explore the possibility of fine-tuning a pre-trained cross-lingual model with only a small number of samples from the target language…
In this paper, we propose a novel deep transfer learning method called deep implicit distribution alignment networks (DIDAN) to deal with cross-corpus speech emotion recognition (SER) problem, in which the labeled training (source) and unlabeled testing (target) speech signals come from different corpora. Specifically, DIDAN first adopts a simple deep regression network consisting of a set of convolutional and fully connected layers to directly regress the source speech spectrums into the emotional labels such that the proposed DIDAN can own the emotion discriminative ability. Then, such abili…
Information Technology University (ITU)-Punjab（旁遮普信息技术大学）；COMSATS University Islamabad (CUI)（伊斯兰堡COMSATS大学）
Cross-lingual speech emotion recognition is an important task for practical applications. The performance of automatic speech emotion recognition systems degrades in cross-corpus scenarios, particularly in scenarios involving multiple languages or a previously unseen language such as Urdu for which limited or no data is available. In this study, we investigate the problem of cross-lingual emotion recognition for Urdu language and contribute URDU---the first ever spontaneous Urdu-language speech emotion database. Evaluations are performed using three different Western languages against Urdu and…
Cross-lingual summarization (CLS) is the task to produce a summary in one particular language for a source document in a different language. We introduce WikiMulti - a new dataset for cross-lingual summarization based on Wikipedia articles in 15 languages. As a set of baselines for further studies, we evaluate the performance of existing cross-lingual abstractive summarization methods on our dataset. We make our dataset publicly available here: https://github.com/tikhonovpavel/wikimulti
University of Southern Queensland（南昆士兰大学）；Information Technology University (ITU)（信息技术大学）；University of the West of England (UWE)（西英格兰大学）
Cross-lingual speech emotion recognition (SER) is a crucial task for many real-world applications. The performance of SER systems is often degraded by the differences in the distributions of training and test data. These differences become more apparent when training and test data belong to different languages, which cause a significant performance gap between the validation and test scores. It is imperative to build more robust models that can fit in practical applications of SER systems. Therefore, in this paper, we propose a Generative Adversarial Network (GAN)-based model for multilingual …
Speaker Style-Aware Phoneme Anchoring for Improved Cross-Lingual Speech Emotion Recognition
Department of Electrical Engineering, National Tsing Hua University, Taiwan（National Tsing Hua University电子工程系）；Language Technologies Institute, Carnegie Mellon University, USA（卡内基梅隆大学语言技术研究所）
University of Sheffield（谢菲尔德大学）；Emotech Labs（Emotech实验室）
For speech emotion datasets, it has been difficult to acquire large quantities of reliable data and acted emotions may be over the top compared to less expressive emotions displayed in everyday life. Lately, larger datasets with natural emotions have been created. Instead of ignoring smaller, acted datasets, this study investigates whether information learnt from acted emotions is useful for detecting natural emotions. Cross-corpus research has mostly considered cross-lingual and even cross-age datasets, and difficulties arise from different methods of annotating emotions causing a drop in per…
Toshiba Europe Limited（东芝欧洲有限公司）；Cambridge Research Laboratory（剑桥研究实验室）
Models that can handle a wide range of speakers and acoustic conditions are essential in speech emotion recognition (SER). Often, these models tend to show mixed results when presented with speakers or acoustic conditions that were not visible during training. This paper investigates the impact of cross-corpus data complementation and data augmentation on the performance of SER models in matched (test-set from same corpus) and mismatched (test-set from different corpus) conditions. Investigations using six emotional speech corpora that include single and multiple speakers as well as variations…
University of Pittsburgh（匹兹堡大学）；Cornell University（康奈尔大学）；aiXplain Inc.（aiXplain公司）
Emotion detection can provide us with a window into understanding human behavior. Due to the complex dynamics of human emotions, however, constructing annotated datasets to train automated models can be expensive. Thus, we explore the efficacy of cross-lingual approaches that would use data from a source language to build models for emotion detection in a target language. We compare three approaches, namely: i) using inherently multilingual models; ii) translating training data into the target language; and iii) using an automatically tagged parallel corpus. In our study, we consider English a…
In the past years, sentiment analysis has increasingly shifted attention to representational frameworks more expressive than semantic polarity (being positive, negative or neutral). However, these richer formats (like Basic Emotions or Valence-Arousal-Dominance, and variants therefrom), rooted in psychological research, tend to proliferate the number of representation schemes for emotion encoding. Thus, a large amount of representationally incompatible emotion lexicons has been developed by various research groups adopting one or the other emotion representation format. As a consequence, the r…
Recent developments in speech emotion recognition (SER) often leverage deep neural networks (DNNs). Comparing and benchmarking different DNN models can often be tedious due to the use of different datasets and evaluation protocols. To facilitate the process, here, we present the Speech Emotion Recognition Adaptation Benchmark (SERAB), a framework for evaluating the performance and generalization capacity of different approaches for utterance-level SER. The benchmark is composed of nine datasets for SER in six languages. Since the datasets have different sizes and numbers of emotional classes, …
UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection
University of West Bohemia, Faculty of Applied Sciences（西波西米亚大学应用科学学院）；Department of Computer Science and Engineering（计算机科学与工程系）；NTIS – New Technologies for the Information Society（信息社会新技术机构）
This article provides a comprehensive investigation on the relations between virality of news articles and the emotions they are found to evoke. Virality, in our view, is a phenomenon with many facets, i.e. under this generic term several different effects of persuasive communication are comprised. By exploiting a high-coverage and bilingual corpus of documents containing metrics of their spread on social networks as well as a massive affective annotation provided by readers, we present a thorough analysis of the interplay between evoked emotions and viral facets. We highlight and discuss our …
University of Salerno（萨勒诺大学）；ICAR-CNR（意大利国家研究委员会信息与高性能计算研究所）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
The development of automated approaches to linguistic acceptability has been greatly fostered by the availability of the English CoLA corpus, which has also been included in the widely used GLUE benchmark. However, this kind of research for languages other than English, as well as the analysis of cross-lingual approaches, has been hindered by the lack of resources with a comparable size in other languages. We have therefore developed the ItaCoLA corpus, containing almost 10,000 sentences with acceptability judgments, which has been created following the same approach and the same steps as the …
Language Technologies Institute, Carnegie Mellon University（语言技术研究所，卡内基梅隆大学）；Athenahealth；Accenture LLP；ARRAY Innovation；Bosch Center for Artificial Intelligence, Bosch Research（博世人工智能中心，博世研究）；Erik Jonsson School of Engineering and Computer Science, University of Texas at Dallas（埃里克·乔纳森工程与计算机科学学院，德克萨斯大学达拉斯分校）；Reality Labs at Meta Platforms, Inc.（Meta平台现实实验室）；Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California (USC)（信号分析与解释实验室（SAIL），明希德电子与计算机工程系，南加州大学（USC））
Technical University of Denmark（丹麦技术大学）；Copenhagen University Hospital（哥本哈根大学医院）；Copenhagen University（哥本哈根大学）
In recent years, speech emotion recognition (SER) has been used in wide ranging applications, from healthcare to the commercial sector. In addition to signal processing approaches, methods for SER now also use deep learning techniques which provide transfer learning possibilities. However, generalizing over languages, corpora and recording conditions is still an open challenge. In this work we address this gap by exploring loss functions that aid in transferability, specifically to non-tonal languages. We propose a variational autoencoder (VAE) with KL annealing and a semi-supervised VAE to ob…
Recent years have seen a rise in interest for cross-lingual transfer between languages with similar typology, and between languages of various scripts. However, the interplay between language similarity and difference in script on cross-lingual transfer is a less studied problem. We explore this interplay on cross-lingual transfer for two supervised tasks, namely part-of-speech tagging and sentiment analysis. We introduce a newly annotated corpus of Algerian user-generated comments comprising parallel annotations of Algerian written in Latin, Arabic, and code-switched scripts, as well as annot…
This work explores the effect of gender and linguistic-based vocal variations on the accuracy of emotive expression classification. Emotive expressions are considered from the perspective of spectral features in speech (Mel-frequency Cepstral Coefficient, Melspectrogram, Spectral Contrast). Emotions are considered from the perspective of Basic Emotion Theory. A convolutional neural network is utilised to classify emotive expressions in emotive audio datasets in English, German, and Italian. Vocal variations for spectral features assessed by (i) a comparative analysis identifying suitable spect…
The success of pretrained cross-lingual language models relies on two essential abilities, i.e., generalization ability for learning downstream tasks in a source language, and cross-lingual transferability for transferring the task knowledge to other languages. However, current methods jointly learn the two abilities in a single-phase cross-lingual pretraining process, resulting in a trade-off between generalization and cross-lingual transfer. In this paper, we propose cross-lingual language model meta-pretraining, which learns the two abilities in different training phases. Our method introdu…
Natural language understanding (NLU) in the context of goal-oriented dialog systems typically includes intent classification and slot labeling tasks. Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to the translated utterances, and thus are sensitive to projection errors. In this work, we propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. We introduce MultiATIS++, a new multilingual NLU corpus that extends the Multilingual ATIS corpus to nine languages …
University of Southern Queensland（南昆士兰大学）；Data61, CSIRO（联邦科学与工业研究组织Data61研究所）；Queensland University of Technology（昆士兰科技大学）；Imperial College London（帝国理工学院）；University of Augsburg（奥格斯堡大学）
Despite the recent advancement in speech emotion recognition (SER) within a single corpus setting, the performance of these SER systems degrades significantly for cross-corpus and cross-language scenarios. The key reason is the lack of generalisation in SER systems towards unseen conditions, which causes them to perform poorly in cross-corpus and cross-language settings. Recent studies focus on utilising adversarial methods to learn domain generalised representation for improving cross-corpus and cross-language SER to address this issue. However, many of these methods only focus on cross-corpu…
University of Edinburgh（爱丁堡大学）；Institute for Language, Cognition and Computation（语言、认知与计算研究所）
We present a cross-lingual summarisation corpus with long documents in a source language associated with multi-sentence summaries in a target language. The corpus covers twelve language pairs and directions for four European languages, namely Czech, English, French and German, and the methodology for its creation can be applied to several other languages. We derive cross-lingual document-summary instances from Wikipedia by combining lead paragraphs and articles' bodies from language aligned Wikipedia titles. We analyse the proposed cross-lingual summarisation task with automatic metrics and va…
In this manuscript, the topic of multi-corpus Speech Emotion Recognition (SER) is approached from a deep transfer learning perspective. A large corpus of emotional speech data, EmoSet, is assembled from a number of existing SER corpora. In total, EmoSet contains 84181 audio recordings from 26 SER corpora with a total duration of over 65 hours. The corpus is then utilised to create a novel framework for multi-corpus speech emotion recognition, namely EmoNet. A combination of a deep ResNet architecture and residual adapters is transferred from the field of multi-domain visual recognition to mult…
University of California, Los Angeles（加州大学洛杉矶分校）；University of Southern California（南加州大学）
Prior work on cross-lingual dependency parsing often focuses on capturing the commonalities between source and target languages and overlooks the potential of leveraging linguistic properties of the languages to facilitate the transfer. In this paper, we show that weak supervisions of linguistic knowledge for the target languages can improve a cross-lingual graph-based dependency parser substantially. Specifically, we explore several types of corpus linguistic statistics and compile them into corpus-wise constraints to guide the inference process during the test time. We adapt two techniques, …
Information Science and Technology（信息科学与技术）；Vidyasirimedhi Institute of Science and Technology（维达亚西姆迪科技研究所）；Chulalongkorn University（朱拉隆功大学）；Department of Dramatic Arts（戏剧艺术系）
University of Helsinki（赫尔辛基大学）；Raamatun Tietokirja, Aika[media] Oy（Raamatun Tietokirja Aika[media]公司）
Twenty-five years ago, morphologically aligned Hebrew-Finnish and Greek-Finnish bitexts (texts accompanied by a translation) were constructed manually in order to create an analytical concordance (Luoto et al., 1997) for a Finnish Bible translation. The creators of the bitexts recently secured the publisher's permission to release its fine-grained alignment, but the alignment was still dependent on proprietary, third-party resources such as a copyrighted text edition and proprietary morphological analyses of the source texts. In this paper, we describe a nontrivial editorial process starting f…
We describe a simple but effective method for cross-lingual syntactic transfer of dependency parsers, in the scenario where a large amount of translation data is not available. The method makes use of three steps: 1) a method for deriving cross-lingual word clusters, which can then be used in a multilingual parser; 2) a method for transferring lexical information from a target language to source language treebanks; 3) a method for integrating these steps with the density-driven annotation projection method of Rasooli and Collins (2015). Experiments show improvements over the state-of-the-art i…
Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation
Zhejiang University（浙江大学）；Westlake University（西湖大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Sichuan Lan-bridge Information Technology Co., Ltd.（四川语言桥信息技术有限公司）；Westlake Institute for Advanced Study（西湖高等研究院）
Motor, Cognitive, or Corpus? What Survives Cross-Lingual Transfer in Speech-Based Parkinsons Disease Detection
Hertie Institute for AI in Brain Health, University of Tübingen（蒂宾根大学赫蒂脑健康人工智能研究所）；Tübingen AI Center, University of Tübingen（蒂宾根大学蒂宾根人工智能中心）；Charité–Universitätsmedizin（夏里特医学院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
JNLP at SemEval-2025 Task 11: Cross-Lingual Multi-Label Emotion Detection Using Generative Models
Japan Advanced Institute of Science and Technology（日本先进科学研究所）；ROIS-DS Center for Juris-Informatics, NII, Tokyo, Japan（日本信息机构ROIS-DS法律信息中心）；National Institute of Advanced Industrial Science and Technology（国家先进工业科学和技术研究所）
University of Southern Queensland（南昆士兰大学）；Data61, CSIRO Australia（澳大利亚联邦科学与工业研究组织Data61）；University of New South Wales（新南威尔士大学）；Queensland University of Technology（昆士兰科技大学）；Imperial College London（帝国理工学院）；University of Augsburg（奥格斯堡大学）
Speech emotion recognition systems (SER) can achieve high accuracy when the training and test data are identically distributed, but this assumption is frequently violated in practice and the performance of SER systems plummet against unforeseen data shifts. The design of robust models for accurate SER is challenging, which limits its use in practical applications. In this paper we propose a deeper neural network architecture wherein we fuse DenseNet, LSTM and Highway Network to learn powerful discriminative features which are robust to noise. We also propose data augmentation with our network …
A Cross-Lingual Meta-Learning Method Based on Domain Adaptation for Speech Emotion Recognition
National Institute for Research and Development in Informatics - ICI Bucharest（布加勒斯特国家信息研究与发展学院）；Academy of Romanian Scientists（罗马尼亚科学院）；University of Bucharest（布加勒斯特大学）；National University of Science and Technology POLITEHNICA Bucharest（布加勒斯特理工国立科技大学）；Paris 1 Panthéon-Sorbonne University（巴黎第一大学先贤祠-索邦大学）
University of São Paulo（圣保罗大学）；Instituto de Matemática e Estatística（数学与统计研究所）
At present, different deep learning models are presenting high accuracy on popular inference datasets such as SNLI, MNLI, and SciTail. However, there are different indicators that those datasets can be exploited by using some simple linguistic patterns. This fact poses difficulties to our understanding of the actual capacity of machine learning models to solve the complex task of textual inference. We propose a new set of syntactic tasks focused on contradiction detection that require specific capacities over linguistic logical forms such as: Boolean coordination, quantifiers, definite descrip…
