Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition
University of Chinese Academy of Sciences（中国科学院大学）；National Key Laboratory of Space Integrated Information System（空间信息集成系统国家重点实验室）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Department of Computer Science and Technology（计算机科学与技术系）
Jingyao Wang, Wenwen Qiang, Changwen Zheng, Fuchun Sun
备注：
This work has been submitted to the IEEE for possible publication
Fine-grained emotion recognition (FER) plays a vital role in various fields, such as disease diagnosis, personalized recommendations, and multimedia mining. However, existing FER methods face three key challenges in real-world applications: (i) they rely on large amounts of continuously annotated data to ensure accuracy since emotions are complex and ambiguous in reality, which is costly and time-consuming; (ii) they cannot capture the temporal heterogeneity caused by changing emotion patterns, because they usually assume that the temporal correlation within sampling periods is the same; (iii) they do not consider the spatial heterogeneity of different FER scenarios, that is, the distribution of emotion information in different data may have bias or interference. To address these challenges, we propose a Spatio-Temporal Fuzzy-oriented Multi-modal Meta-learning framework (ST-F2M). Specifically, ST-F2M first divides the multi-modal videos into multiple views, and each view corresponds to one modality of one emotion. Multiple randomly selected views for the same emotion form a meta-training task. Next, ST-F2M uses an integrated module with spatial and temporal convolutions to encode the data of each task, reflecting the spatial and temporal heterogeneity. Then it adds fuzzy semantic information to each task based on generalized fuzzy rules, which helps handle the complexity and ambiguity of emotions. Finally, ST-F2M learns emotion-related general meta-knowledge through meta-recurrent neural networks to achieve fast and robust fine-grained emotion recognition. Extensive experiments show that ST-F2M outperforms various state-of-the-art methods in terms of accuracy and model efficiency. In addition, we construct ablation studies and further analysis to explore why ST-F2M performs well.
School of Electronics Engineering, Kyungpook National University（庆北国立大学电子工程学院）
Generating music with emotion similar to that of an input video is a very relevant issue nowadays. Video content creators and automatic movie directors benefit from maintaining their viewers engaged, which can be facilitated by producing novel material eliciting stronger emotions in them. Moreover, there's currently a demand for more empathetic computers to aid humans in applications such as augmenting the perception ability of visually and/or hearing impaired people. Current approaches overlook the video's emotional characteristics in the music generation step, only consider static images ins…
École de Technologie Supérieure, Université du Québec（魁北克大学高等技术学院）
Facial expressions are one of the most powerful ways for depicting specific patterns in human behavior and describing human emotional state. Despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot handle properly variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task even for humans. In this paper, we investigate the suitability of state-of-the-art deep learning architectures based on con…
University of San Diego（圣地亚哥大学）；University of California San Diego（加州大学圣迭戈分校）
In this paper, we introduce an extension of our presented cognitive-based emotion model [27][28]and [30], where we enhance our knowledge-based emotion unit of the architecture by embedding a fuzzy rule-based system to it. The model utilizes the cognitive parameters dependency and their corresponding weights to regulate the robot's behavior and fuse their behavior data to achieve the final decision in their interaction with the environment. Using this fuzzy system, our previous model can simulate linguistic parameters for better controlling and generating understandable and flexible behaviors i…
Our experiment adapts several popular deep learning methods as well as some traditional methods on the problem of video emotion recognition. In our experiment, we use the CNN-LSTM architecture for visual information extraction and classification and utilize traditional methods such as for audio feature classification. For multimodal fusion, we use the traditional Support Vector Machine. Our experiment yields a good result on the AFEW 6.0 Dataset.
We perform spatio-temporal analysis of public sentiment using geotagged photo collections. We develop a deep learning-based classifier that predicts the emotion conveyed by an image. This allows us to associate sentiment with place. We perform spatial hotspot detection and show that different emotions have distinct spatial distributions that match expectations. We also perform temporal analysis using the capture time of the photos. Our spatio-temporal hotspot detection correctly identifies emerging concentrations of specific emotions and year-by-year analyses of select locations show there are…
NVIDIA（英伟达）；Samsung AI Center（三星人工智能中心）；University of Athens（雅典大学）；Imperial College London（帝国理工学院）
Training deep neural networks with spatio-temporal (i.e., 3D) or multidimensional convolutions of higher-order is computationally challenging due to millions of unknown parameters across dozens of layers. To alleviate this, one approach is to apply low-rank tensor decompositions to convolution kernels in order to compress the network and reduce its number of parameters. Alternatively, new convolutional blocks, such as MobileNet, can be directly designed for efficiency. In this paper, we unify these two approaches by proposing a tensor factorization framework for efficient multidimensional (sep…
Region-aware Spatiotemporal Modeling with Collaborative Domain Generalization for Cross-Subject EEG Emotion Recognition
Laboratory of Digital Image and Intelligent Computation, Shanghai Maritime University, Shanghai 201306, China（上海海洋大学数字图像与智能计算实验室）；Department of Language Science and Technology, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China（香港理工大学语言科学与技术系）；School of Chinese as a Second Language, and Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China（北京大学第二语言学院及MRI研究中心）；CAS Key Laboratory of Behavioral Science, Center for Brain Science and Learning Difficulties, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China（中国科学院行为科学重点实验室）；Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing 100875, China（北京应用实验心理学重点实验室）；Institute of Computing and Intelligence, Harbin Institute of Technology Shenzhen, Shenzhen 518000, China（哈尔滨工业大学深圳校区计算机与智能研究所）
Chongqing University（重庆大学）；Nanning Normal University（南宁师范大学）；University of Technology Sydney（悉尼科技大学）
Gait emotion recognition plays a crucial role in the intelligent system. Most of the existing methods recognize emotions by focusing on local actions over time. However, they ignore that the effective distances of different emotions in the time domain are different, and the local actions during walking are quite similar. Thus, emotions should be represented by global states instead of indirect local actions. To address these issues, a novel Multi Scale Adaptive Graph Convolution Network (MSA-GCN) is presented in this work through constructing dynamic temporal receptive fields and designing mul…
CGTGait: Collaborative Graph and Transformer for Gait Emotion Recognition
Huazhong University of Science and Technology, China（华中科技大学）；Wuhan University of Technology, China（武汉理工大学）；Hefei University of Technology, China（合肥工业大学）；Boston University, United States（波士顿大学）
Université de Montréal（蒙特利尔大学）；Goethe-Universität Frankfurt（法兰克福歌德大学）
The task of the emotion recognition in the wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies. The videos depict acted-out emotions under realistic conditions with a large degree of variation in attributes such as pose and illumination, making it worthwhile to explore approaches which consider combinations of features from multiple modalities for label assignment. In this paper we present our approach to learning several specialist models using deep learning techniques, each focusing on one modality. Among these are a convoluti…
Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition
Russian - Armenian University（俄罗斯-亚美尼亚大学）；Sber AI Lab（Sber AI实验室）；HSE University（俄罗斯高等经济大学）；Innopolis University（Innopolis大学）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院可信人工智能研究中心）
iCV Research Group, Institute of Technology, University of Tartu（塔尔图大学技术研究所 iCV 研究组）；Computer Vision Center（计算机视觉中心）；University of Barcelona（巴塞罗那大学）；University of Autonoma, Barcelona（巴塞罗那自治大学）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）；Institute of Physiology and Pathology of Hearing（听力生理学与病理学研究所）；Department of Psychology, University of Tartu（塔尔图大学心理学系）；The Estonian Center of Behavioral and Health Sciences（爱沙尼亚行为与健康科学中心）；Department of Electrical and Electronic Engineering, Hasan Kalyoncu University（哈桑·卡尔永库大学电气与电子工程系）
Humans modify their facial expressions in order to communicate their internal states and sometimes to mislead observers regarding their true emotional states. Evidence in experimental psychology shows that discriminative facial responses are short and subtle. This suggests that such behavior would be easier to distinguish when captured in high resolution at an increased frame rate. We are proposing SASE-FE, the first dataset of facial expressions that are either congruent or incongruent with underlying emotion states. We show that overall the problem of recognizing whether facial movements are…
Recent studies demonstrated that the average heart rate (HR) can be measured from facial videos based on non-contact remote photoplethysmography (rPPG). However for many medical applications (e.g., atrial fibrillation (AF) detection) knowing only the average HR is not sufficient, and measuring precise rPPG signals from face for heart rate variability (HRV) analysis is needed. Here we propose an rPPG measurement method, which is the first work to use deep spatio-temporal networks for reconstructing precise rPPG signals from raw facial videos. With the constraint of trend-consistency with ground…
Single-Stream Multi-Feature Fusion with Temporal Robustness for Gait Emotion Recognition
College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；College of Physical Science and Technology, Southwest University（西南大学物理科学与技术学院）；Wisesoft Inc.（四川川大智胜软件股份有限公司）
Speech sounds of spoken language are obtained by varying configuration of the articulators surrounding the vocal tract. They contain abundant information that can be utilized to better understand the underlying mechanism of human speech production. We propose a novel deep neural network-based learning framework that understands acoustic information in the variable-length sequence of vocal tract shaping during speech production, captured by real-time magnetic resonance imaging (rtMRI), and translate it into text. The proposed framework comprises of spatiotemporal convolutions, a recurrent netwo…
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）
Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions
LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada（ETS蒙特利尔大学系统工程系LIVIA实验室）；LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada（ETS蒙特利尔大学软件与信息工程系LIVIA实验室）；Dept. of Health, Kinesiology, & Applied Physiology, Concordia University, Montreal, Canada（康科迪亚大学健康、运动科学与应用生理学系）；Montreal Behavioural Medicine Centre, CIUSSS Nord-de-l’Ile-de-Montréal, Canada（蒙特利尔行为医学中心，蒙特利尔北岛卫生与社会服务局）
University of St Andrews（圣安德鲁斯大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Oulu（奥卢大学）
Being spontaneous, micro-expressions are useful in the inference of a person's true emotions even if an attempt is made to conceal them. Due to their short duration and low intensity, the recognition of micro-expressions is a difficult task in affective computing. The early work based on handcrafted spatio-temporal features which showed some promise, has recently been superseded by different deep learning approaches which now compete for the state of the art performance. Nevertheless, the problem of capturing both local and global spatio-temporal patterns remains challenging. To this end, here…
University of Miami（迈阿密大学）；Kalinga Institute of Industrial Technology（卡林加工业技术学院）；JP Morgan Chase & Co（摩根大通公司）；Walmart Global Tech（沃尔玛全球技术）
Automated human emotion recognition from facial expressions is a well-studied problem and still remains a very challenging task. Some efficient or accurate deep learning models have been presented in the literature. However, it is quite difficult to design a model that is both efficient and accurate at the same time. Moreover, identifying the minute feature variations in facial regions for both macro and micro-expressions requires expertise in network design. In this paper, we proposed to search for a highly efficient and robust neural architecture for both macro and micro-level facial express…
University of South Florida（南佛罗里达大学）；USF Health（南佛罗里达大学健康科学中心）
The current practice for assessing neonatal postoperative pain relies on bedside caregivers. This practice is subjective, inconsistent, slow, and discontinuous. To develop a reliable medical interpretation, several automated approaches have been proposed to enhance the current practice. These approaches are unimodal and focus mainly on assessing neonatal procedural (acute) pain. As pain is a multimodal emotion that is often expressed through multiple modalities, the multimodal assessment of pain is necessary especially in case of postoperative (acute prolonged) pain. Additionally, spatio-tempo…
We describe a content based video retrieval (CBVR) software system for identifying specific locations of a human action within a full length film, and retrieving similar video shots from a query. For this, we introduce the concept of a trajectory point cloud for classifying unique actions, encoded in a spatio-temporal covariant eigenspace, where each point is characterized by its spatial location, local Frenet-Serret vector basis, time averaged curvature and torsion and the mean osculating hyperplane. Since each action can be distinguished by their unique trajectories within this space, the tr…
IIIT Sri City（印度信息技术学院斯里城分校）；The University of Chicago（芝加哥大学）；Shiv Nadar University（希夫·纳达尔大学）
Micro-expression has emerged as a promising modality in affective computing due to its high objectivity in emotion detection. Despite the higher recognition accuracy provided by the deep learning models, there are still significant scope for improvements in micro-expression recognition techniques. The presence of micro-expressions in small-local regions of the face, as well as the limited size of available databases, continue to limit the accuracy in recognizing micro-expressions. In this work, we propose a facial micro-expression recognition model using 3D residual attention network named MER…
Feng Chia University（逢甲大学）；NTUNHS（台湾新竹护理健康大学）；Xiamen University Malaysia（厦门大学马来西亚分校）；Xiamen University（厦门大学）
Owing to the development and advancement of artificial intelligence, numerous works were established in the human facial expression recognition system. Meanwhile, the detection and classification of micro-expressions are attracting attentions from various research communities in the recent few years. In this paper, we first review the processes of a conventional optical-flow-based recognition system, which comprised of facial landmarks annotations, optical flow guided images computation, features extraction and emotion class categorization. Secondly, a few approaches have been proposed to impr…
Chongqing School, University of Chinese Academy of Sciences（中国科学院大学重庆学院）；University College London（伦敦大学学院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；Southwest University（西南大学）
As a spontaneous expression of emotion on face, micro-expression reveals the underlying emotion that cannot be controlled by human. In micro-expression, facial movement is transient and sparsely localized through time. However, the existing representation based on various deep learning techniques learned from a full video clip is usually redundant. In addition, methods utilizing the single apex frame of each video clip require expert annotations and sacrifice the temporal dynamics. To simultaneously localize and recognize such fleeting facial movements, we propose a novel end-to-end deep learn…
Benchmarking Micro-action Recognition: Dataset, Methods, and Applications
Hefei University of Technology (HFUT)（合肥工业大学）；Hefei Comprehensive National Science Center, Institute of Artificial Intelligence（合肥综合性国家科学中心人工智能研究院）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）；Lanzhou University（兰州大学）
DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Computer Science and Technology, Southwest University of Science and Technology（西南科技大学计算机科学与技术学院）；School of Data Science, University of Science and Technology of China（中国科学技术大学数据科学学院）
Key Lab of Intelligent Information Processing, Chinese Academy of Sciences (CAS) Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）；School of Computer Science and Technology University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）
This report describes a multi-modal multi-task ($M^3$T) approach underlying our submission to the valence-arousal estimation track of the Affective Behavior Analysis in-the-wild (ABAW) Challenge, held in conjunction with the IEEE International Conference on Automatic Face and Gesture Recognition (FG) 2020. In the proposed $M^3$T framework, we fuse both visual features from videos and acoustic features from the audio tracks to estimate the valence and arousal. The spatio-temporal visual features are extracted with a 3D convolutional network and a bidirectional recurrent neural network. Consider…
Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data
School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院，合肥工业大学）；Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
