Generative Technology for Human Emotion Recognition: A Scope Review
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology (GZ)（香港科技大学（广州））；Beijing Normal University（北京师范大学）；Shenzhen University（深圳大学）；University of Electronic Science and Technology of China（电子科技大学）
Fei Ma, Yucheng Yuan, Yifan Xie, Hongwei Ren, Ivan Liu, Ying He, Fuji Ren, Fei Richard Yu, Shiguang Ni
情感计算处于人工智能（AI）的前沿，旨在赋予机器理解并回应人类情感的能力。该领域的核心是情感识别，它致力于从语音、面部图像、文本和生理信号等不同模态中识别和解释人类情感状态。近年来，生成模型取得了重要进展，包括自编码器（Autoencoder）、生成对抗网络（Generative Adversarial Network）、扩散模型（Diffusion Model）和大语言模型（Large Language Model）。这些模型凭借其强大的数据生成能力，成为推动情感识别发展的关键工具。然而，迄今为止，系统性地综述面向情感识别的生成技术的工作仍然匮乏。本综述旨在通过对截至2024年6月的320余篇研究论文进行全面分析，填补现有文献中的空白。具体而言，本综述将首先介绍不同生成模型的数学原理和常用数据集。随后，通过一种分类体系，深入分析生成技术如何基于不同模态在数据增强、特征提取、半监督学习、跨域等多个方面解决情感识别问题。最后，本综述将概述未来研究方向，强调生成模型在推动情感识别领域发展和增强AI系统情感智能方面的潜力。
英文摘要：
Affective computing stands at the forefront of artificial intelligence (AI), seeking to imbue machines with the ability to comprehend and respond to human emotions. Central to this field is emotion recognition, which endeavors to identify and interpret human emotional states from different modalities, such as speech, facial images, text, and physiological signals. In recent years, important progress has been made in generative models, including Autoencoder, Generative Adversarial Network, Diffusion Model, and Large Language Model. These models, with their powerful data generation capabilities, emerge as pivotal tools in advancing emotion recognition. However, up to now, there remains a paucity of systematic efforts that review generative technology for emotion recognition. This survey aims to bridge the gaps in the existing literature by conducting a comprehensive analysis of over 320 research papers until June 2024. Specifically, this survey will firstly introduce the mathematical principles of different generative models and the commonly used datasets. Subsequently, through a taxonomy, it will provide an in-depth analysis of how generative techniques address emotion recognition based on different modalities in several aspects, including data augmentation, feature extraction, semi-supervised learning, cross-domain, etc. Finally, the review will outline future research directions, emphasizing the potential of generative models to advance the field of emotion recognition and enhance the emotional intelligence of AI systems.
A Review of Human Emotion Synthesis Based on Generative Technology
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；MICS Thrust, The Hong Kong University of Science and Technology (GZ)（香港科技大学（广州）MICS学科）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）；School of Information Technology, Carleton University（卡尔顿大学信息技术学院）
Humans are able to comprehend information from multiple domains for e.g. speech, text and visual. With advancement of deep learning technology there has been significant improvement of speech recognition. Recognizing emotion from speech is important aspect and with deep learning technology emotion recognition has improved in accuracy and latency. There are still many challenges to improve accuracy. In this work, we attempt to explore different neural networks to improve accuracy of emotion recognition. With different architectures explored, we find (CNN+RNN) + 3DCNN multi-model architecture wh…
Emotion-Aware Interaction Design in Intelligent User Interface Using Multi-Modal Deep Learning
Carnegie Mellon University（卡内基梅隆大学）；University of Maryland（马里兰大学）；School of Visual Arts（视觉艺术学院）；New York University（纽约大学）；University of Southern California（南加州大学）
LTCI, Télécom-Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI实验室）；Direction Innovation & Recherche SNCF（法国国家铁路公司创新与研究部）
Several recent studies on dyadic human-human interactions have been done on conversations without specific business objectives. However, many companies might benefit from studies dedicated to more precise environments such as after sales services or customer satisfaction surveys. In this work, we place ourselves in the scope of a live chat customer service in which we want to detect emotions and their evolution in the conversation flow. This context leads to multiple challenges that range from exploiting restricted, small and mostly unlabeled datasets to finding and adapting methods for such c…
Technidoo Solutions Lab（Technidoo Solutions实验室）；Mongolian University of Science and Technology（蒙古科技大学）
We proposed the industry level deep learning approach for speech emotion recognition task. In industry, carefully proposed deep transfer learning technology shows real results due to mostly low amount of training data availability, machine training cost, and specialized learning on dedicated AI tasks. The proposed speech recognition framework, called DeepEMO, consists of two main pipelines such that preprocessing to extract efficient main features and deep transfer learning model to train and recognize. Main source code is in https://github.com/enkhtogtokh/deepemo repository
University of Edinburgh（爱丁堡大学）；Jozef Stefan Institute（约瑟夫·斯特凡研究所）
Research in automatic affect recognition has seldom addressed the issue of computational resource utilization. With the advent of ambient intelligence technology which employs a variety of low-power, resource-constrained devices, this issue is increasingly gaining interest. This is especially the case in the context of health and elderly care technologies, where interventions may rely on monitoring of emotional status to provide support or alert carers as appropriate. This paper focuses on emotion recognition from speech data, in settings where it is desirable to minimize memory and computatio…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Amazon Alexa（亚马逊Alexa）
Speech emotion recognition (SER) is a key technology to enable more natural human-machine communication. However, SER has long suffered from a lack of public large-scale labeled datasets. To circumvent this problem, we investigate how unsupervised representation learning on unlabeled datasets can benefit SER. We show that the contrastive predictive coding (CPC) method can learn salient representations from unlabeled datasets, which improves emotion recognition performance. In our experiments, this method achieved state-of-the-art concordance correlation coefficient (CCC) performance for all em…
Recognizing the patient's emotions using deep learning techniques has attracted significant attention recently due to technological advancements. Automatically identifying the emotions can help build smart healthcare centers that can detect depression and stress among the patients in order to start the medication early. Using advanced technology to identify emotions is one of the most exciting topics as it defines the relationships between humans and machines. Machines learned how to predict emotions by adopting various methods. In this survey, we present recent research in the field of using …
Indian Institute of Information Technology, Allahabad（阿拉哈巴德印度信息技术学院）
To enable humanoid robots to share our social space we need to develop technology for easy interaction with the robots using multiple modes such as speech, gestures and share our emotions with them. We have targeted this research towards addressing the core issue of emotion recognition problem which would require less computation resources and much lesser number of network hyperparameters which will be more adaptive to be computed on low resourced social robots for real time communication. More specifically, here we have proposed an Inception module based Convolutional Neural Network Architect…
Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Zhengzhou University of Light Industry（郑州轻工业大学）；Nanjing University（南京大学）；Aalto University（阿尔托大学）；Beijing Institute of Technology（北京理工大学）
Emotion recognition technology through analyzing the EEG signal is currently an essential concept in Artificial Intelligence and holds great potential in emotional health care, human-computer interaction, multimedia content recommendation, etc. Though there have been several works devoted to reviewing EEG-based emotion recognition, the content of these reviews needs to be updated. In addition, those works are either fragmented in content or only focus on specific techniques adopted in this area but neglect the holistic perspective of the entire technical routes. Hence, in this paper, we review…
Deep Learning Based Approach to Enhanced Recognition of Emotions and Behavioral Patterns of Autistic Children
Department of Information Technology（信息科技系）；Sri Lanka Institute of Information Technology（斯里兰卡信息科技研究所）；Department of Computer Systems Engineering（计算机系统工程系）
