Recent studies show that depression can be partially reflected from human facial attributes. Since facial attributes have various data structure and carry different information, existing approaches fail to specifically consider the optimal way to extract depression-related features from each of them, as well as investigates the best fusion strategy. In this paper, we propose to extend Neural Architecture Search (NAS) technique for designing an optimal model for multiple facial attributes-based depression recognition, which can be efficiently and robustly implemented in a small dataset. Our approach first conducts a warmer up step to the feature extractor of each facial attribute, aiming to largely reduce the search space and providing customized architecture, where each feature extractor can be either a Convolution Neural Networks (CNN) or Graph Neural Networks (GNN). Then, we conduct an end-to-end architecture search for all feature extractors and the fusion network, allowing the complementary depression cues to be optimally combined with less redundancy. The experimental results on AVEC 2016 dataset show that the model explored by our approach achieves breakthrough performance with 27\% and 30\% RMSE and MAE improvements over the existing state-of-the-art. In light of these findings, this paper provides solid evidences and a strong baseline for applying NAS to time-series data-based mental health analysis.
Harbin University of Science and Technology（哈尔滨理工大学）；University of Cambridge（剑桥大学）；University of Nottingham（诺丁汉大学）
Video-based automatic depression analysis provides a fast, objective and repeatable self-assessment solution, which has been widely developed in recent years. While depression clues may be reflected by human facial behaviours of various temporal scales, most existing approaches either focused on modelling depression from short-term or video-level facial behaviours. In this sense, we propose a two-stage framework that models depression severity from multi-scale short-term and video-level facial behaviours. The short-term depressive behaviour modelling stage first deep learns depression-related …
MF-GCN: A Multi-Frequency Graph Convolutional Network for Tri-Modal Depression Detection Using Eye-Tracking, Facial, and Acoustic Features
Department of Computer Science, New Uzbekistan University, Tashkent, Uzbekistan（计算机科学系，新乌兹别克斯坦大学）；Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh（机器人与机电工程系，达卡大学）；Department of Electrical and Computer Engineering, University of Virginia, USA（电气与计算机工程系，弗吉尼亚大学）
Faces of the Mind: Unveiling Mental Health States Through Facial Expressions in 11,427 Adolescents
The Affiliated Brain Hospital of Nanjing Medical University, and the School of Biomedical Engineering and Informatics of Nanjing Medical University（南京医科大学附属脑科医院及南京医科大学生物医学工程与信息学院）
FacialPulse: An Efficient RNN-based Depression Detection via Temporal Facial Landmarks
Hefei University of Technology（合肥工业大学）；CFAR（计算科学与工程中心）；IHPC（高性能计算研究所）；A*STAR（新加坡科技研究局）；University of Science and Technology of China（中国科学技术大学）；The University of Electro-Communications（东京电机大学）
Université Paris-Est（巴黎东部大学）；INSAT Institut National des Sciences Appliquées et de Technologie（突尼斯应用科学与技术学院）；Yobitrust（优比特信托公司）；University of Sciences and Arts in Lebanon（黎巴嫩科学与艺术大学）；Université de Bourgogne Franche Comté（勃艮第-弗朗什-孔泰大学）
Clinical depression or Major Depressive Disorder (MDD) is a common and serious medical illness. In this paper, a deep recurrent neural network-based framework is presented to detect depression and to predict its severity level from speech. Low-level and high-level audio features are extracted from audio recordings to predict the 24 scores of the Patient Health Questionnaire and the binary class of depression diagnosis. To overcome the problem of the small size of Speech Depression Recognition (SDR) datasets, expanding training labels and transferred features are considered. The proposed approa…
University of Oulu（奥卢大学）；VTT Technical Research Centre of Finland（芬兰VTT技术研究中心）
Deep learning models have shown promising results in recognizing depressive states using video-based facial expressions. While successful models typically leverage using 3D-CNNs or video distillation techniques, the different use of pretraining, data augmentation, preprocessing, and optimization techniques across experiments makes it difficult to make fair architectural comparisons. We propose instead to enhance two simple models based on ResNet-50 that use only static spatial information by using two specific face alignment methods and improved data augmentation, optimization, and scheduling …
Harbin Engineering University（哈尔滨工程大学）；Suzhou Fanhan Information Technology Company（苏州凡汉信息技术公司）；Sun Yat-sen University（中山大学）；Southern Marine Science and Engineering Guangdong Laboratory (Zhanjiang)（南方海洋科学与工程广东省实验室（湛江））；Marine Biomedical Research Institute of Guangdong（广东海洋生物医药研究院）；Harvard Medical School（哈佛医学院）
The Self-Rating Depression Scale (SDS) questionnaire is commonly utilized for effective depression preliminary screening. The uncontrolled self-administered measure, on the other hand, maybe readily influenced by insouciant or dishonest responses, yielding different findings from the clinician-administered diagnostic. Facial expression (FE) and behaviors are important in clinician-administered assessments, but they are underappreciated in self-administered evaluations. We use a new dataset of 200 participants to demonstrate the validity of self-rating questionnaires and their accompanying ques…
Exploring Facial Biomarkers for Detecting Depression through Temporal Analysis of Action Units
Machine Learning and Data Analytics Lab, Department Artificial Intelligence in Biomedical Engineering (AIBE), Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（机器学习与数据分析实验室，人工智能生物医学工程部门（AIBE），弗里德里希-亚历山大-埃森纳赫大学（FAU））；Department of Clinical Psychology and Psychotherapy, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（临床心理学与心理治疗系，弗里德里希-亚历山大-埃森纳赫大学（FAU））；Chair of Visual Computing, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（视觉计算教授，计算机科学系，弗里德里希-亚历山大-埃森纳赫大学（FAU））；Translational Digital Health Group, Institute of AI for Health, Helmholtz Zentrum München – German Research Center for Environmental Health（转化数字健康组，人工智能与健康研究所，海德堡中心——德国环境健康研究研究中心）
A Deep Learning approach for Depressive Symptoms assessment in Parkinson's disease patients using facial videos
Foundation for Research and Technology Hellas (FORTH)（希腊研究与技术基金会）；University of Crete（克里特大学）；University Hospital of Heraklion（伊拉克利翁大学医院）；Patras University Hospital（帕特雷大学医院）；University of Ioannina（约阿尼纳大学）
Early detection of psychological distress is key to effective treatment. Automatic detection of distress, such as depression, is an active area of research. Current approaches utilise vocal, facial, and bodily modalities. Of these, the bodily modality is the least investigated, partially due to the difficulty in extracting bodily representations from videos, and partially due to the lack of viable datasets. Existing body modality approaches use automatic categorization of expressions to represent body language as a series of specific expressions, much like words within natural language. In thi…
With more than 300 million people depressed worldwide, depression is a global problem. Due to access barriers such as social stigma, cost, and treatment availability, 60% of mentally-ill adults do not receive any mental health services. Effective and efficient diagnosis relies on detecting clinical symptoms of depression. Automatic detection of depressive symptoms would potentially improve diagnostic accuracy and availability, leading to faster intervention. In this work, we present a machine learning method for measuring the severity of depressive symptoms. Our multi-modal method uses 3D faci…
University of Oulu（奥卢大学）；VTT Technical Research Centre of Finland Ltd（芬兰VTT技术研究中心有限公司）
Depression is a mental illness that may be harmful to an individual's health. The detection of mental health disorders in the early stages and a precise diagnosis are critical to avoid social, physiological, or psychological side effects. This work analyzes physiological signals to observe if different depressive states have a noticeable impact on the blood volume pulse (BVP) and the heart rate variability (HRV) response. Although typically, HRV features are calculated from biosignals obtained with contact-based sensors such as wearables, we propose instead a novel scheme that directly extract…
Center for Machine Vision and Signal Analysis, University of Oulu（奥卢大学机器视觉与信号分析中心）；Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Remote photoplethysmography (rPPG), which aims at measuring heart activities without any contact, has great potential in many applications (e.g., remote healthcare). Existing end-to-end rPPG and heart rate (HR) measurement methods from facial videos are vulnerable to the less-constrained scenarios (e.g., with head movement and bad illumination). In this letter, we explore the reason why existing end-to-end networks perform poorly in challenging conditions and establish a strong end-to-end baseline (AutoHR) for remote HR measurement with neural architecture search (NAS). The proposed method inc…
University of Oulu（奥卢大学）；École de Technologie Supérieure（高等技术学院）；VTT Technical Research Centre of Finland Ltd（芬兰VTT技术研究中心有限公司）
Video-based analysis of facial expressions has been increasingly applied to infer health states of individuals, such as depression and pain. Among the existing approaches, deep learning models composed of structures for multiscale spatiotemporal processing have shown strong potential for encoding facial dynamics. However, such models have high computational complexity, making for a difficult deployment of these solutions. To address this issue, we introduce a new technique to decompose the extraction of multiscale spatiotemporal features. Particularly, a building block structure called Decompo…
University of Washington（华盛顿大学）；Texas A&M University（德克萨斯A&M大学）
We present our preliminary work to determine if patient's vocal acoustic, linguistic, and facial patterns could predict clinical ratings of depression severity, namely Patient Health Questionnaire depression scale (PHQ-8). We proposed a multi modal fusion model that combines three different modalities: audio, video , and text features. By training over AVEC 2017 data set, our proposed model outperforms each single modality prediction model, and surpasses the data set baseline with ice margin.
Depression is a major debilitating disorder which can affect people from all ages. With a continuous increase in the number of annual cases of depression, there is a need to develop automatic techniques for the detection of the presence and extent of depression. In this AVEC challenge we explore different modalities (speech, language and visual features extracted from face) to design and develop automatic methods for the detection of depression. In psychology literature, the PHQ-8 questionnaire is well established as a tool for measuring the severity of depression. In this paper we aim to auto…
Context Matters: Vision-Based Depression Detection Comparing Classical and Deep Approaches
Intelligent Systems Program, University of Pittsburgh（匹兹堡大学智能系统项目）；CGI Technologies and Solutions Inc（CGI技术与解决方案公司）；University of Oregon（俄勒冈大学）；Department of Health Information Management, University of Pittsburgh（匹兹堡大学健康信息管理系）；Utrecht University（乌得勒支大学）；Department of Psychology, University of Pittsburgh（匹兹堡大学心理学系）
Department of Systems Design Engineering（系统设计工程系）；Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）；University of Waterloo（滑铁卢大学）
A core challenge faced by the majority of individuals with Autism Spectrum Disorder (ASD) is an impaired ability to infer other people's emotions based on their facial expressions. With significant recent advances in machine learning, one potential approach to leveraging technology to assist such individuals to better recognize facial expressions and reduce the risk of possible loneliness and depression due to social isolation is the design of computer vision-driven facial expression recognition systems. Motivated by this social need as well as the low latency requirement of such systems, this…
Facial micro-expressions indicate brief and subtle facial movements that appear during emotional communication. In comparison to macro-expressions, micro-expressions are more challenging to be analyzed due to the short span of time and the fine-grained changes. In recent years, micro-expression recognition (MER) has drawn much attention because it can benefit a wide range of applications, e.g. police interrogation, clinical diagnosis, depression analysis, and business negotiation. In this survey, we offer a fresh overview to discuss new research directions and challenges these days for MER tas…
Just Another Media Laboratory (JAM Lab)（另一个媒体实验室（JAM Lab））
We present a new facial recognition system, capable of identifying a person, provided their likeness has been previously stored in the system, in real time. The system is based on storing and comparing facial embeddings of the subject, and identifying them later within a live video feed. This system is highly accurate, and is able to tag people with their ID in real time. It is able to do so, even when using a database containing thousands of facial embeddings, by using a parallelized searching technique. This makes the system quite fast and allows it to be highly scalable.
Encoded Local Projections (ELP) is a recently introduced dense sampling image descriptor which uses projections in small neighbourhoods to construct a histogram/descriptor for the entire image. ELP has shown to be as accurate as other state-of-the-art features in searching medical images while being time and resource efficient. This paper attempts for the first time to utilize ELP descriptor as primary features for facial recognition and compare the results with LBP histogram on the Labeled Faces in the Wild dataset. We have evaluated descriptors by comparing the chi-squared distance of each i…
Content based image retrieval, a technique which uses visual contents of image to search images from large scale image databases according to users' interests. This paper provides a comprehensive survey on recent technology used in the area of content based face image retrieval. Nowadays digital devices and photo sharing sites are getting more popularity, large human face photos are available in database. Multiple types of facial features are used to represent discriminality on large scale human facial image database. Searching and mining of facial images are challenging problems and important…
Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data
LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada（LIVIA，系统工程系，蒙特利尔大学工程学院，加拿大）；LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada（LIVIA，软件与信息技术工程系，蒙特利尔大学工程学院，加拿大）；Dept. of Health, Kinesiology & Applied Physiology, Concordia University, Montreal, Canada（健康、运动科学与应用生理学系，康科迪亚大学，蒙特利尔，加拿大）
Escuela Superior Politecnica del Litoral (ESPOL)（滨海高等理工学院（ESPOL））
Face recall is a basic human cognitive process performed routinely, e.g., when meeting someone and determining if we have met that person before. Assisting a subject during face recall by suggesting candidate faces can be challenging. One of the reasons is that the search space - the face space - is quite large and lacks structure. A commercial application of face recall is facial composite systems - such as Identikit, PhotoFIT, and CD-FIT - where a witness searches for an image of a face that resembles his memory of a particular offender. The inherent uncertainty and cost in the evaluation of…
