Source-free domain adaptation in visual emotion recognition (SFDA-VER) is a highly challenging task that requires adapting VER models to the target domain without relying on source data, which is of great significance for data privacy protection. However, due to the unignorable disparities between visual emotion data and traditional image classification data, existing SFDA methods perform poorly on this task. In this paper, we investigate the SFDA-VER task from a fuzzy perspective and identify two key issues: fuzzy emotion labels and fuzzy pseudo-labels. These issues arise from the inherent uncertainty of emotion annotations and the potential mispredictions in pseudo-labels. To address these issues, we propose a novel fuzzy-aware loss (FAL) to enable the VER model to better learn and adapt to new domains under fuzzy labels. Specifically, FAL modifies the standard cross entropy loss and focuses on adjusting the losses of non-predicted categories, which prevents a large number of uncertain or incorrect predictions from overwhelming the VER model during adaptation. In addition, we provide a theoretical analysis of FAL and prove its robustness in handling the noise in generated pseudo-labels. Extensive experiments on 26 domain adaptation sub-tasks across three benchmark datasets demonstrate the effectiveness of our method.
Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition
University of Chinese Academy of Sciences（中国科学院大学）；National Key Laboratory of Space Integrated Information System（空间信息集成系统国家重点实验室）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Department of Computer Science and Technology（计算机科学与技术系）
Politecnico di Milano（米兰理工大学）；Beihang University（北京航空航天大学）
Online damage quantification suffers from insufficient labeled data that weakens its accuracy. In this context, adopting the domain adaptation on historical labeled data from similar structures/damages or simulated digital twin data to assist the current diagnosis task would be beneficial. However, most domain adaptation methods are designed for classification and cannot efficiently address damage quantification, a regression problem with continuous real-valued labels. This study first proposes a novel domain adaptation method, the Online Fuzzy-set-based Joint Distribution Adaptation for Regre…
Speech emotion recognition plays an important role in building more intelligent and human-like agents. Due to the difficulty of collecting speech emotional data, an increasingly popular solution is leveraging a related and rich source corpus to help address the target corpus. However, domain shift between the corpora poses a serious challenge, making domain shift adaptation difficult to function even on the recognition of positive/negative emotions. In this work, we propose class-wise adversarial domain adaptation to address this challenge by reducing the shift for all classes between differen…
PRISM: Prioritized Channel Importance with Semi-supervised Domain Adaptation for Cross-Subject EEG Emotion Recognition
School of Computing, T. J Watson College of Engineering and Applied Science, Binghamton University - State University of New York（计算学院、T.J Watson工程与应用科学学院、宾夕法尼亚州立大学宾夕法尼亚州立大学）
Tsinghua University（清华大学）；Nankai University（南开大学）；University of California, Berkeley（加州大学伯克利分校）；Didi Chuxing（滴滴出行）
Thanks to large-scale labeled training data, deep neural networks (DNNs) have obtained remarkable success in many vision and multimedia tasks. However, because of the presence of domain shift, the learned knowledge of the well-trained DNNs cannot be well generalized to new domains or datasets that have few labels. Unsupervised domain adaptation (UDA) studies the problem of transferring models trained on one labeled source domain to another unlabeled target domain. In this paper, we focus on UDA in visual emotion analysis for both emotion distribution learning and dominant emotion classificatio…
HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）；Anhui University（安徽大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
As an essential element for the diagnosis and rehabilitation of psychiatric disorders, the electroencephalogram (EEG) based emotion recognition has achieved significant progress due to its high precision and reliability. However, one obstacle to practicality lies in the variability between subjects and sessions. Although several studies have adopted domain adaptation (DA) approaches to tackle this problem, most of them treat multiple EEG data from different subjects and sessions together as a single source domain for transfer, which either fails to satisfy the assumption of domain adaptation t…
EvoFA: Evolvable Fast Adaptation for EEG Emotion Recognition
Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学生命与健康产业研究院）；University of Hong Kong（香港大学）；Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所类脑智能研究中心）
College of Information Engineering, Zhejiang University of Technology（浙江工业大学信息工程学院）；Institute of Wenzhou, Zhejiang University（浙江大学温州市研究院）；College of Mathematical Medicine, Zhejiang Normal University（浙江师范大学数学医学学院）；Key Laboratory for Biomedical Engineering of MOE of China, Department of Biomedical Engineering, Zhejiang University（中国教育部生物医学工程重点实验室，浙江大学生物医学工程学院）
Emotion recognition from speech is one of the key steps towards emotional intelligence in advanced human-machine interaction. Identifying emotions in human speech requires learning features that are robust and discriminative across diverse domains that differ in terms of language, spontaneity of speech, recording conditions, and types of emotions. This corresponds to a learning scenario in which the joint distributions of features and labels may change substantially across domains. In this paper, we propose a deep architecture that jointly exploits a convolutional network for extracting domain…
M3D: Manifold-based Domain Adaptation with Dynamic Distribution for Non-Deep Transfer Learning in Cross-subject and Cross-session EEG-based Emotion Recognition
School of Biomedical Engineering, Medical School, Shenzhen University（生物医学工程学院，医学学院，深圳大学）；School of Public Health, Shenzhen University Medical School（公共卫生学院，深圳大学医学学院）；Department of Radiology, Nanshan Hospital of Shenzhen University（放射科，深圳大学南山医院）；School of Mathematical Sciences, Shenzhen University（数学科学学院，深圳大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（申明武智能工程学院，华南理工大学）；Institute for Super Robotics, Guangzhou, Guangdong, China（超机器人研究所，广州，广东，中国）
iFuzzyTL: Interpretable Fuzzy Transfer Learning for SSVEP BCI System
University of Technology Sydney（悉尼科技大学）；Australian AI Institute（澳大利亚人工智能研究院）；School of Computer Science, Faculty of Engineering and Information Technology（工程与信息技术学院计算机科学学院）
With the improved performance of deep learning, the number of studies trying to apply deep learning to human emotion analysis is increasing rapidly. But even with this trend going on, it is still difficult to obtain high-quality images and annotations. For this reason, the Learning from Synthetic Data (LSD) Challenge, which learns from synthetic images and infers from real images, is one of the most interesting areas. In general, Domain Adaptation methods are widely used to address LSD challenges, but there is a limitation that target domains (real images) are still needed. Focusing on these l…
Emotion and expressivity in music have been topics of considerable interest in the field of music information retrieval. In recent years, mid-level perceptual features have been suggested as means to explain computational predictions of musical emotion. We find that the diversity of musical styles and genres in the available dataset for learning these features is not sufficient for models to generalise well to specialised acoustic domains such as solo piano music. In this work, we show that by utilising unsupervised domain adaptation together with receptive-field regularised deep neural networ…
Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（上海人工智能教育研究院，华东师范大学，中国）；School of Computer Science and Technology, East China Normal University, China（计算机科学与技术学院，华东师范大学，中国）；Boin Hearing Technology (Shanghai) Co., LTD, China（博英听力技术（上海）有限公司，中国）；Department of Computing Science, University of Aberdeen, UK（计算科学系，阿伯丁大学，英国）
Speech-based machine learning (ML) has been heralded as a promising solution for tracking prosodic and spectrotemporal patterns in real-life that are indicative of emotional changes, providing a valuable window into one's cognitive and mental state. Yet, the scarcity of labelled data in ambulatory studies prevents the reliable training of ML models, which usually rely on "data-hungry" distribution-based learning. Leveraging the abundance of labelled speech data from acted emotions, this paper proposes a few-shot learning approach for automatically recognizing emotion in spontaneous speech from…
School of Electronics Engineering, Kyungpook National University（庆北国立大学电子工程学院）
Generating music with emotion similar to that of an input video is a very relevant issue nowadays. Video content creators and automatic movie directors benefit from maintaining their viewers engaged, which can be facilitated by producing novel material eliciting stronger emotions in them. Moreover, there's currently a demand for more empathetic computers to aid humans in applications such as augmenting the perception ability of visually and/or hearing impaired people. Current approaches overlook the video's emotional characteristics in the music generation step, only consider static images ins…
University of Southern California（南加利福尼亚大学）；National Technical University of Athens（雅典国家技术大学）
The study of Music Cognition and neural responses to music has been invaluable in understanding human emotions. Brain signals, though, manifest a highly complex structure that makes processing and retrieving meaningful features challenging, particularly of abstract constructs like affect. Moreover, the performance of learning models is undermined by the limited amount of available neuronal data and their severe inter-subject variability. In this paper we extract efficient, personalized affective representations from EEG signals during music listening. To this end, we employ music signals as a …
PAII Inc.（PAII公司）；Imperial College London（帝国理工学院）；Tsinghua University（清华大学）；Nankai University（南开大学）
Emotions are usually evoked in humans by images. Recently, extensive research efforts have been dedicated to understanding the emotions of images. In this chapter, we aim to introduce image emotion analysis (IEA) from a computational perspective with the focus on summarizing recent advances and suggesting future directions. We begin with commonly used emotion representation models from psychology. We then define the key computational problems that the researchers have been trying to solve and provide supervised frameworks that are generally used for different IEA tasks. After the introduction …
Zhejiang University（浙江大学）；HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）
Electroencephalography (EEG) based emotion recognition has demonstrated tremendous improvement in recent years. Specifically, numerous domain adaptation (DA) algorithms have been exploited in the past five years to enhance the generalization of emotion recognition models across subjects. The DA methods assume that calibration data (although unlabeled) exists in the target domain (new user). However, this assumption conflicts with the application scenario that the model should be deployed without the time-consuming calibration experiments. We argue that domain generalization (DG) is more reason…
Robustness to environmental noise is important to creating automatic speech emotion recognition systems that are deployable in the real world. Prior work on noise robustness has assumed that systems would not make use of sample-by-sample training noise conditions, or that they would have access to unlabelled testing data to generalize across noise conditions. We avoid these assumptions and introduce the resulting task as heterogeneous condition training. We show that with full knowledge of the test noise conditions, we can improve performance by dynamically routing samples to specialized featu…
Columbia University（哥伦比亚大学）；Recognition Technologies, Inc.（识别技术公司）
This paper presents a transfer learning method in speech emotion recognition based on a Time-Delay Neural Network (TDNN) architecture. A major challenge in the current speech-based emotion detection research is data scarcity. The proposed method resolves this problem by applying transfer learning techniques in order to leverage data from the automatic speech recognition (ASR) task for which ample data is available. Our experiments also show the advantage of speaker-class adaptation modeling techniques by adopting identity-vector (i-vector) based features in addition to standard Mel-Frequency C…
University of San Diego（圣地亚哥大学）；University of California San Diego（加州大学圣迭戈分校）
In this paper, we introduce an extension of our presented cognitive-based emotion model [27][28]and [30], where we enhance our knowledge-based emotion unit of the architecture by embedding a fuzzy rule-based system to it. The model utilizes the cognitive parameters dependency and their corresponding weights to regulate the robot's behavior and fuse their behavior data to achieve the final decision in their interaction with the environment. Using this fuzzy system, our previous model can simulate linguistic parameters for better controlling and generating understandable and flexible behaviors i…
Multi-modal Speech Emotion Recognition via Feature Distribution Adaptation Network
School of Computer and Control Engineering, Yantai University（烟台大学计算机与控制工程学院）；Business School, Xuzhou University of Technology（徐州工程学院商学院）；College of Computer Science, Nankai University（南开大学计算机学院）；Southeast University（东南大学）
Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony
Wolf 1069B Lab, Sany Group（沃尔夫1069B实验室，三一集团）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Engineering and Technology, University of Washington, Tacoma（华盛顿大学塔科姆分校工程与科技学院）；Alibaba Group（阿里巴巴集团）；Department of Engineering, King’s College London（伦敦大学国王学院工程系）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
