Automatic Micro-Expression Apex Frame Spotting using Local Binary Pattern from Six Intersection Planes
University of Tabriz（大不里士大学）；Qazvin Branch, Islamic Azad University（伊斯兰阿扎德大学加兹温分校）
Vida Esmaeili, Mahmood Mohassel Feghhi, Seyed Omid Shahdi
备注：
6 pages, 7 figures, Presented at the 11 Iranian and the first International Conference on Machine Vision and Image Processing (MVIP), 19-20 February 2020, https://mvip2020.ut.ac.ir/paper?manu=39079
Facial expressions are one of the most effective ways for non-verbal communications, which can be expressed as the Micro-Expression (ME) in the high-stake situations. The MEs are involuntary, rapid, and, subtle, and they can reveal real human intentions. However, their feature extraction is very challenging due to their low intensity and very short duration. Although Local Binary Pattern from Three Orthogonal Plane (LBP-TOP) feature extractor is useful for the ME analysis, it does not consider essential information. To address this problem, we propose a new feature extractor called Local Binary Pattern from Six Intersection Planes (LBP-SIPl). This method extracts LBP code on six intersection planes, and then it combines them. Results show that the proposed method has superior performance in apex frame spotting automatically in comparison with the relevant methods on the CASME database. Simulation results show that, using the proposed method, the apex frame has been spotted in 43% of subjects in the CASME database, automatically. Also, the mean absolute error of 1.76 is achieved, using our novel proposed method.
Faculty of Information Technology and Electrical Engineering, University of Oulu（奥卢大学信息技术与电气工程学院）
Micro-expressions are rapid and involuntary facial expressions, which indicate the suppressed or concealed emotions. Recently, the research on automatic micro-expression (ME) spotting obtains increasing attention. ME spotting is a crucial step prior to further ME analysis tasks. The spotting results can be used as important cues to assist many other human-oriented tasks and thus have many potential applications. In this paper, by investigating existing ME spotting methods, we recognize the immediacy of standardizing the performance evaluation of micro-expression spotting methods. To this end, …
Institute of Psychology, Chinese Academy of Sciences（中国科学院心理研究所）；University of the Chinese Academy of Sciences（中国科学院大学）；Manchester Metropolitan University（曼彻斯特城市大学）
This paper presents baseline results for the Third Facial Micro-Expression Grand Challenge (MEGC 2020). Both macro- and micro-expression intervals in CAS(ME)$^2$ and SAMM Long Videos are spotted by employing the method of Main Directional Maximal Difference Analysis (MDMD). The MDMD method uses the magnitude maximal difference in the main direction of optical flow features to spot facial movements. The single-frame prediction results of the original MDMD method are post-processed into reasonable video intervals. The metric F1-scores of baseline results are evaluated: for CAS(ME)$^2$, the F1-sc…
Kalyani Government Engineering College（卡利亚尼政府工程学院）；Nanyang Technological University（南洋理工大学）；Cognizant Technology Solutions（高知特科技解决方案公司）；Indian Institute of Technology Roorkee（印度理工学院鲁尔基分校）
Facial micro-expressions are sudden involuntary minute muscle movements which reveal true emotions that people try to conceal. Spotting a micro-expression and recognizing it is a major challenge owing to its short duration and intensity. Many works pursued traditional and deep learning based approaches to solve this issue but compromised on learning low-level features and higher accuracy due to unavailability of datasets. This motivated us to propose a novel joint architecture of spatial and temporal network which extracts time-contrasted features from the feature maps to contrast out micro-ex…
CentraleSupélec（中央理工-高等电力学院）；CNRS（法国国家科学研究中心）；IETR（雷恩电子与电信研究所）；Institute of Psychology, Chinese Academy of Sciences（中国科学院心理研究所）；Manchester Metropolitan University（曼彻斯特城市大学）
This paper presents baseline results for the first Micro-Expression Spotting Challenge 2019 by evaluating local temporal pattern (LTP) on SAMM and CAS(ME)2. The proposed LTP patterns are extracted by applying PCA in a temporal window on several facial local regions. The micro-expression sequences are then spotted by a local classification of LTP and a global fusion. The performance is evaluated by Leave-One-Subject-Out cross validation. Furthermore, we define the criteria of determining true positives in one video by overlap rate and set the metric F1-score for spotting performance of the whol…
University of Oulu（奥卢大学）；Xi’an Jiaotong University（西安交通大学）
Micro-expressions (MEs) are brief and involuntary facial expressions that occur when people are trying to hide their true feelings or conceal their emotions. Based on psychology research, MEs play an important role in understanding genuine emotions, which leads to many potential applications. Therefore, ME analysis has become an attractive topic for various research areas, such as psychology, law enforcement, and psychotherapy. In the computer vision field, the study of MEs can be divided into two main tasks, spotting and recognition, which are used to identify positions of MEs in videos and d…
University of Oulu（奥卢大学）；University of Oxford（牛津大学）
Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these ch…
Synergistic Spotting and Recognition of Micro-Expression via Temporal State Transition
School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）
Micro-expressions (MEs) are infrequent and uncontrollable facial events that can highlight emotional deception and appear in a high-stakes environment. This paper propose an algorithm for spatiotemporal MEs spotting. Since MEs are unusual events, we treat them as abnormal patterns that diverge from expected Normal Facial Behaviour (NFBs) patterns. NFBs correspond to facial muscle activations, eye blink/gaze events and mouth opening/closing movements that are all facial deformation but not MEs. We propose a probabilistic model to estimate the probability density function that models the spatiot…
University of Science and Technology of China（中国科学技术大学）
Automatic Micro-Expression (ME) spotting in long videos is a crucial step in ME analysis but also a challenging task due to the short duration and low intensity of MEs. When solving this problem, previous works generally lack in considering the structures of human faces and the correspondence between expressions and relevant facial muscles. To address this issue for better performance of ME spotting, this paper seeks to extract finer spatial features by modeling the relationships between facial Regions of Interest (ROIs). Specifically, we propose a graph convolutional-based network, called Act…
Intelligent Security Center（智能安全中心）；UCL Interaction Centre（伦敦大学学院交互中心）；University College London（伦敦大学学院）；Southwest University（西南大学）
The automatic recognition of micro-expression has been boosted ever since the successful introduction of deep learning approaches. As researchers working on such topics are moving to learn from the nature of micro-expression, the practice of using deep learning techniques has evolved from processing the entire video clip of micro-expression to the recognition on apex frame. Using the apex frame is able to get rid of redundant video frames, but the relevant temporal evidence of micro-expression would be thereby left out. This paper proposes a novel Apex-Time Network (ATNet) to recognize micro-e…
Multimedia University（多媒体大学）；Heriot-Watt University Malaysia（赫瑞-瓦特大学马来西亚校区）
Facial expressions vary from the visible to the subtle. In recent years, the analysis of micro-expressions $-$ a natural occurrence resulting from the suppression of one's true emotions, has drawn the attention of researchers with a broad range of potential applications. However, spotting microexpressions in long videos becomes increasingly challenging when intertwined with normal or macro-expressions. In this paper, we propose a shallow optical flow three-stream CNN (SOFTNet) model to predict a score that captures the likelihood of a frame being in an expression interval. By fashioning the sp…
Manchester Metropolitan University（曼彻斯特城市大学）；University of Manchester（曼彻斯特大学）
Facial expression spotting is the preliminary step for micro- and macro-expression analysis. The task of reliably spotting such expressions in video sequences is currently unsolved. The current best systems depend upon optical flow methods to extract regional motion features, before categorisation of that motion into a specific class of facial movement. Optical flow is susceptible to drift error, which introduces a serious problem for motions with long-term dependencies, such as high frame-rate macro-expression. We propose a purely deep learning solution which, rather than tracking frame diffe…
Feng Chia University（逢甲大学）；Multimedia University（多媒体大学）；Monash University Malaysia（马来西亚莫纳什大学）
Despite recent interest and advances in facial micro-expression research, there is still plenty room for improvement in terms of micro-expression recognition. Conventional feature extraction approaches for micro-expression video consider either the whole video sequence or a part of it, for representation. However, with the high-speed video capture of micro-expressions (100-200 fps), are all frames necessary to provide a sufficiently meaningful representation? Is the luxury of data a bane to accurate recognition? A novel proposition is presented in this paper, whereby we utilize only two images…
With the growth of popularity of facial micro-expressions in recent years, the demand for long videos with micro- and macro-expressions remains high. Extended from SAMM, a micro-expressions dataset released in 2016, this paper presents SAMM Long Videos dataset for spontaneous micro- and macro-expressions recognition and spotting. SAMM Long Videos dataset consists of 147 long videos with 343 macro-expressions and 159 micro-expressions. The dataset is FACS-coded with detailed Action Units (AUs). We compare our dataset with Chinese Academy of Sciences Macro-Expressions and Micro-Expressions (CAS(…
MELDAE: A Framework for Micro-Expression Spotting, Detection, and Automatic Evaluation in In-the-Wild Conversational Scenes
College of Computer Science, National University of Defense Technology, China（国防科技大学计算机科学学院）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, China（华南理工大学智能工程学院）；College of Computer Science and Technology, Zhejiang University, China（浙江大学计算机科学与技术学院）；College of Computer Science and Software Engineering, Shenzhen University, China（深圳大学计算机科学与软件工程学院）
Feng Chia University（逢甲大学）；Xiamen University Malaysia（厦门大学马来西亚分校）；Xinyang Normal University（信阳师范学院）；University Teknologi Malaysia Kuala Lumpur（马来西亚理工大学吉隆坡分校）
When a person attempts to conceal an emotion, the genuine emotion is manifest as a micro-expression. Exploration of automatic facial micro-expression recognition systems is relatively new in the computer vision domain. This is due to the difficulty in implementing optimal feature extraction methods to cope with the subtlety and brief motion characteristics of the expression. Most of the existing approaches extract the subtle facial movements based on hand-crafted features. In this paper, we address the micro-expression recognition task with a convolutional neural network (CNN) architecture, wh…
Facial micro-expressions indicate brief and subtle facial movements that appear during emotional communication. In comparison to macro-expressions, micro-expressions are more challenging to be analyzed due to the short span of time and the fine-grained changes. In recent years, micro-expression recognition (MER) has drawn much attention because it can benefit a wide range of applications, e.g. police interrogation, clinical diagnosis, depression analysis, and business negotiation. In this survey, we offer a fresh overview to discuss new research directions and challenges these days for MER tas…
Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has rec…
MEGC2025: Micro-Expression Grand Challenge on Spot Then Recognize and Visual Question Answering
Department of Computing and Mathematics, Manchester Metropolitan University（计算与数学系，曼彻斯特 Metropolitan 大学）；State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences（认知科学与心理健康国家重点实验室，心理学研究所，中国科学院）；Department of Psychology, University of the Chinese Academy of Sciences（心理学系，中国科学院大学）；National Taiwan University（台湾大学）；Zhejiang University（浙江大学）；University of Oulu（奥卢大学）；Harbin Institute of Technology（哈尔滨工业大学）
Micro-Expression Recognition by Motion Feature Extraction based on Pre-training
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；School of Medicine and Biological Information Engineering, Northeastern University（东北大学医学与生物信息工程学院）；Shenyang Contain Electronic Technology Co., Ltd.（沈阳 Contain 电子科技有限公司）
STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition
Vision Intelligence and Machine Learning (VIML) Group, School of Computing and Electrical Engineering, Indian Institute of Technology Mandi（印度理工学院曼迪分校计算与电气工程学院视觉智能与机器学习（VIML）实验室）；Indian Institute of Information Technology Bhagalpur（印度信息技术学院巴加尔普尔分校）
Manchester Metropolitan University（曼彻斯特城市大学）；Panasonic R&D Center（松下研发中心）；Emotional Intelligence Academy（情商学院）
Micro-facial expressions are regarded as an important human behavioural event that can highlight emotional deception. Spotting these movements is difficult for humans and machines, however research into using computer vision to detect subtle facial expressions is growing in popularity. This paper proposes an individualised baseline micro-movement detection method using 3D Histogram of Oriented Gradients (3D HOG) temporal difference method. We define a face template consisting of 26 regions based on the Facial Action Coding System (FACS). We extract the temporal features of each region using 3D…
Multimedia University（多媒体大学）；University of Nottingham（诺丁汉大学）；Monash University Malaysia（莫纳什大学马来西亚分校）
Over the last few years, automatic facial micro-expression analysis has garnered increasing attention from experts across different disciplines because of its potential applications in various fields such as clinical diagnosis, forensic investigation and security systems. Advances in computer algorithms and video acquisition technology have rendered machine analysis of facial micro-expressions possible today, in contrast to decades ago when it was primarily the domain of psychiatrists where analysis was largely manual. Indeed, although the study of facial micro-expressions is a well-establishe…
Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Micro-expressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-express…
Chongqing School, University of Chinese Academy of Sciences（中国科学院大学重庆学院）；University College London（伦敦大学学院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；Southwest University（西南大学）
As a spontaneous expression of emotion on face, micro-expression reveals the underlying emotion that cannot be controlled by human. In micro-expression, facial movement is transient and sparsely localized through time. However, the existing representation based on various deep learning techniques learned from a full video clip is usually redundant. In addition, methods utilizing the single apex frame of each video clip require expert annotations and sacrifice the temporal dynamics. To simultaneously localize and recognize such fleeting facial movements, we propose a novel end-to-end deep learn…
Feng Chia University（逢甲大学）；National Taipei University of Nursing and Health Sciences（台北护理健康大学）；Multimedia University（多媒体大学）；Xinyang Normal University（信阳师范大学）
In the recent year, state-of-the-art for facial micro-expression recognition have been significantly advanced by deep neural networks. The robustness of deep learning has yielded promising performance beyond that of traditional handcrafted approaches. Most works in literature emphasized on increasing the depth of networks and employing highly complex objective functions to learn more features. In this paper, we design a Shallow Triple Stream Three-dimensional CNN (STSTNet) that is computationally light whilst capable of extracting discriminative high level features and details of micro-express…
