Micro-Expression (ME) is the spontaneous, involuntary movement of a face that can reveal the true feeling. Recently, increasing researches have paid attention to this field combing deep learning techniques. Action units (AUs) are the fundamental actions reflecting the facial muscle movements and AU detection has been adopted by many researches to classify facial expressions. However, the time-consuming annotation process makes it difficult to correlate the combinations of AUs to specific emotion classes. Inspired by the nodes relationship building Graph Convolutional Networks (GCN), we propose an end-to-end AU-oriented graph classification network, namely MER-GCN, which uses 3D ConvNets to extract AU features and applies GCN layers to discover the dependency laying between AU nodes for ME categorization. To our best knowledge, this work is the first end-to-end architecture for Micro-Expression Recognition (MER) using AUs based GCN. The experimental results show that our approach outperforms CNN-based MER networks.
Hikvision Research Institute（海康威视研究院）；College of Information Science & Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
Graph Convolutional Networks (GCNs) have attracted increasing interests for the task of skeleton-based action recognition. The key lies in the design of the graph structure, which encodes skeleton topology information. In this paper, we propose Dynamic GCN, in which a novel convolutional neural network named Contextencoding Network (CeN) is introduced to learn skeleton topology automatically. In particular, when learning the dependency between two joints, contextual features from the rest joints are incorporated in a global manner. CeN is extremely lightweight yet effective, and can be embedde…
Convolutional Neural Networks (CNNs) achieve impressive performance in a wide variety of fields. Their success benefited from a massive boost when very deep CNN models were able to be reliably trained. Despite their merits, CNNs fail to properly address problems with non-Euclidean data. To overcome this challenge, Graph Convolutional Networks (GCNs) build graphs to represent non-Euclidean data, borrow concepts from CNNs, and apply them in training. GCNs show promising results, but they are usually limited to very shallow models due to the vanishing gradient problem. As a result, most state-of-…
National University of Singapore（新加坡国立大学）；Taipei Medical University（台北医学大学）；Nanyang Technological University（南洋理工大学）；Wan Fang Hospital（万芳医院）；Institute of Data Science（数据科学研究所）
The morphology and distribution of microcalcifications in a cluster are the most important characteristics for radiologists to diagnose breast cancer. However, it is time-consuming and difficult for radiologists to identify these characteristics, and there also lacks of effective solutions for automatic characterization. In this study, we proposed a multi-task deep graph convolutional network (GCN) method for the automatic characterization of morphology and distribution of microcalcifications in mammograms. Our proposed method transforms morphology and distribution characterization into node a…
Autonomous driving vehicles (ADVs) hold great hopes to solve traffic congestion problems and reduce the number of traffic accidents. Accurate trajectories prediction of other traffic agents around ADVs is of key importance to achieve safe and efficient driving. Pedestrians, particularly, are more challenging to forecast due to their complex social in-teractions and randomly moving patterns. We propose a Residual Graph Convolutional Neural Network (Res-GCNN), which models the interactive behaviors of pedes-trians by using the adjacent matrix of the constructed graph for the current scene. Thoug…
University of Science and Technology of China（中国科学技术大学）
Automatic Micro-Expression (ME) spotting in long videos is a crucial step in ME analysis but also a challenging task due to the short duration and low intensity of MEs. When solving this problem, previous works generally lack in considering the structures of human faces and the correspondence between expressions and relevant facial muscles. To address this issue for better performance of ME spotting, this paper seeks to extract finer spatial features by modeling the relationships between facial Regions of Interest (ROIs). Specifically, we propose a graph convolutional-based network, called Act…
University of Mannheim（曼海姆大学）；Carnegie Mellon University（卡内基梅隆大学）；Alibaba Group（阿里巴巴集团）；The University of Hong Kong（香港大学）；University of Siegen（锡根大学）
Skeleton-based action recognition aims to recognize human actions given human joint coordinates with skeletal interconnections. By defining a graph with joints as vertices and their natural connections as edges, previous works successfully adopted Graph Convolutional networks (GCNs) to model joint co-occurrences and achieved superior performance. More recently, a limitation of GCNs is identified, i.e., the topology is fixed after training. To relax such a restriction, Self-Attention (SA) mechanism has been adopted to make the topology of GCNs adaptive to the input, resulting in the state-of-th…
School of Computer Science and Communication Engineering, Jiangsu University（江苏大学计算机科学与通信工程学院）；Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）
Micro-Expression Recognition (MER) is a challenging task as the subtle changes occur over different action regions of a face. Changes in facial action regions are formed as Action Units (AUs), and AUs in micro-expressions can be seen as the actors in cooperative group activities. In this paper, we propose a novel deep neural network model for objective class-based MER, which simultaneously detects AUs and aggregates AU-level features into micro-expression-level representation through Graph Convolutional Networks (GCN). Specifically, we propose two new strategies in our AU detection module for …
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）
Cancer prognostication is a challenging task in computational pathology that requires context-aware representations of histology features to adequately infer patient survival. Despite the advancements made in weakly-supervised deep learning, many approaches are not context-aware and are unable to model important morphological feature interactions between cell identities and tissue types that are prognostic for patient survival. In this work, we present Patch-GCN, a context-aware, spatially-resolved patch-based graph convolutional network that hierarchically aggregates instance-level histology …
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）
Skeleton-based human action recognition has attracted much attention with the prevalence of accessible depth sensors. Recently, graph convolutional networks (GCNs) have been widely used for this task due to their powerful capability to model graph data. The topology of the adjacency graph is a key factor for modeling the correlations of the input skeletons. Thus, previous methods mainly focus on the design/learning of the graph topology. But once the topology is learned, only a single-scale feature and one transformation exist in each layer of the networks. Many insights, such as multi-scale i…
An Empirical Study of Super-resolution on Low-resolution Micro-expression Recognition
Macau University of Science and Technology（澳门科技大学）；Southeast University（东南大学）；Nanjing Institute of Technology（南京工程学院）；Jiangsu University（江苏大学）；University of Oulu（奥卢大学）
University of Miami（迈阿密大学）；Indian Institute of Technology Ropar（印度理工学院罗巴尔分校）；Malaviya National Institute of Technology（马拉维亚国家理工学院）
Micro expression recognition (MER) is a very challenging area of research due to its intrinsic nature and fine-grained changes. In the literature, the problem of MER has been solved through handcrafted/descriptor-based techniques. However, in recent times, deep learning (DL) based techniques have been adopted to gain higher performance for MER. Also, rich survey articles on MER are available by summarizing the datasets, experimental settings, conventional and deep learning methods. In contrast, these studies lack the ability to convey the impact of network design paradigms and experimental set…
Image captioning is shown to be able to achieve a better performance by using scene graphs to represent the relations of objects in the image. The current captioning encoders generally use a Graph Convolutional Net (GCN) to represent the relation information and merge it with the object region features via concatenation or convolution to get the final input for sentence decoding. However, the GCN-based encoders in the existing methods are less effective for captioning due to two reasons. First, using the image captioning as the objective (i.e., Maximum Likelihood Estimation) rather than a rela…
IIIT Sri City（印度信息技术学院斯里城分校）；The University of Chicago（芝加哥大学）；Shiv Nadar University（希夫·纳达尔大学）
Micro-expression has emerged as a promising modality in affective computing due to its high objectivity in emotion detection. Despite the higher recognition accuracy provided by the deep learning models, there are still significant scope for improvements in micro-expression recognition techniques. The presence of micro-expressions in small-local regions of the face, as well as the limited size of available databases, continue to limit the accuracy in recognizing micro-expressions. In this work, we propose a facial micro-expression recognition model using 3D residual attention network named MER…
Nanjing University of Science and Technology（南京理工大学）；Shanghai Jiao Tong University（上海交通大学）；Monash University（莫纳什大学）
Nowadays, deep learning methods, especially the Graph Convolutional Network (GCN), have shown impressive performance in hyperspectral image (HSI) classification. However, the current GCN-based methods treat graph construction and image classification as two separate tasks, which often results in suboptimal performance. Another defect of these methods is that they mainly focus on modeling the local pairwise importance between graph nodes while lack the capability to capture the global contextual information of HSI. In this paper, we propose a Multi-level GCN with Automatic Graph Learning method…
Peking University（北京大学）；Deepwise AI Lab（深睿人工智能实验室）
Due to its safety-critical property, the image-based diagnosis is desired to achieve robustness on out-of-distribution (OOD) samples. A natural way towards this goal is capturing only clinically disease-related features, which is composed of macroscopic attributes (e.g., margins, shapes) and microscopic image-based features (e.g., textures) of lesion-related areas. However, such disease-related features are often interweaved with data-dependent (but disease irrelevant) biases during learning, disabling the OOD generalization. To resolve this problem, we propose a novel framework, namely Domain…
MPT: Motion Prompt Tuning for Micro-Expression Recognition
Key Laboratory of Child Development and Learning Science of Ministry of Education, Southeast University（教育部儿童发展与学习科学重点实验室，东南大学）；School of Biological Science and Medical Engineering, Southeast University（生物科学与医学工程学院，东南大学）；School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；Australian Institute for Machine Learning, the University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；School of Computer Science and Technology, China University of Mining and Mine Digitization Engineering Research Center of the Ministry of Education（计算机科学与技术学院，教育部矿业数字化工程研究中心，中国矿业大学）；Department of Data Science & AI, Monash University（数据科学与人工智能系，墨尔本大学）
FAMNet: Integrating 2D and 3D Features for Micro-expression Recognition via Multi-task Learning and Hierarchical Attention
School of Software, Northwestern Polytechnical University（软件学院，西北工业大学）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；School of Cyber Science and Engineering, Zhengzhou University（网络科学与工程学院，郑州大学）
Brigham and Women's Hospital（布莱根和妇女医院）；Harvard Medical School（哈佛医学院）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）
Histology-based grade classification is clinically important for many cancer types in stratifying patients distinct treatment groups. In prostate cancer, the Gleason score is a grading system used to measure the aggressiveness of prostate cancer from the spatial organization of cells and the distribution of glands. However, the subjective interpretation of Gleason score often suffers from large interobserver and intraobserver variability. Previous work in deep learning-based objective Gleason grading requires manual pixel-level annotation. In this work, we propose a weakly-supervised approach …
Southeast University（东南大学）；South China University of Technology（华南理工大学）；University of Oulu（奥卢大学）；Nanjing University of Science and Technology（南京理工大学）
Cross-database micro-expression recognition (CDMER) is one of recently emerging and interesting problem in micro-expression analysis. CDMER is more challenging than the conventional micro-expression recognition (MER), because the training and testing samples in CDMER come from different micro-expression databases, resulting in the inconsistency of the feature distributions between the training and testing sets. In this paper, we contribute to this topic from three aspects. First, we establish a CDMER experimental evaluation protocol aiming to allow the researchers to conveniently work on this …
University of Oulu（奥卢大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Haaga-Helia University of Applied Sciences（哈格-赫利尔应用科学大学）
Micro-expressions (MEs) are involuntary facial movements revealing people's hidden feelings in high-stake situations and have practical importance in medical treatment, national security, interrogations and many human-computer interaction systems. Early methods for MER mainly based on traditional appearance and geometry features. Recently, with the success of deep learning (DL) in various fields, neural networks have received increasing interests in MER. Different from macro-expressions, MEs are spontaneous, subtle, and rapid facial movements, leading to difficult data collection, thus have sm…
DA-Flow: Dual Attention Normalizing Flow for Skeleton-based Video Anomaly Detection
School of Information and Communication Engineering, University of Electronic Science and Technology of China (UESTC)（电子科技大学信息与通信工程学院）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Laboratoire des Signaux et Systemes, Universite Paris-Saclay, CNRS, CentraleSupelec（巴黎萨克雷大学信号与系统实验室、法国国家科学研究中心、中央理工学院）
Rethinking Key-frame-based Micro-expression Recognition: A Robust and Accurate Framework Against Key-frame Errors
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Auckland（奥克兰大学）；Key Laboratory of Interactive Technology and Experience System, Ministry of Culture and Tourism（文化和旅游部交互技术与体验系统重点实验室）
AHMSA-Net: Adaptive Hierarchical Multi-Scale Attention Network for Micro-Expression Recognition
College of Computer Science and Technology（计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education（教育部符号计算与知识工程重点实验室）；Jilin University（吉林大学）
Facial micro-expressions indicate brief and subtle facial movements that appear during emotional communication. In comparison to macro-expressions, micro-expressions are more challenging to be analyzed due to the short span of time and the fine-grained changes. In recent years, micro-expression recognition (MER) has drawn much attention because it can benefit a wide range of applications, e.g. police interrogation, clinical diagnosis, depression analysis, and business negotiation. In this survey, we offer a fresh overview to discuss new research directions and challenges these days for MER tas…
DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Computer Science and Technology, Southwest University of Science and Technology（西南科技大学计算机科学与技术学院）；School of Data Science, University of Science and Technology of China（中国科学技术大学数据科学学院）
Jiangsu University（江苏大学）；Southeast University（东南大学）；Nanjing Institute of Technology（南京工程学院）
Micro-expression recognition (\textbf{MER}) has attracted lots of researchers' attention in a decade. However, occlusion will occur for MER in real-world scenarios. This paper deeply investigates an interesting but unexplored challenging issue in MER, \ie, occlusion MER. First, to research MER under real-world occlusion, synthetic occluded micro-expression databases are created by using various mask for the community. Second, to suppress the influence of occlusion, a \underline{R}egion-inspired \underline{R}elation \underline{R}easoning \underline{N}etwork (\textbf{RRRN}) is proposed to model …
Learning Mutual Excitation for Hand-to-Hand and Human-to-Human Interaction Recognition
State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能省重点实验室）；OPPO AI Center（OPPO人工智能中心）；Sony R&D Center China（索尼中国研发中心）；Peng Cheng Laboratory（鹏城实验室）
Nanjing University of Posts and Telecommunications（南京邮电大学）；University of Oulu（奥卢大学）；Stanford University（斯坦福大学）
Micro-expression recognition (MER) is valuable because micro-expressions (MEs) can reveal genuine emotions. Most works take image sequences as input and cannot effectively explore ME information because subtle ME-related motions are easily submerged in unrelated information. Instead, the facial landmark is a low-dimensional and compact modality, which achieves lower computational cost and potentially concentrates on ME-related movement features. However, the discriminability of facial landmarks for MER is unclear. Thus, this paper explores the contribution of facial landmarks and proposes a no…
