Current day pain assessment methods rely on patient self-report or by an observer like the Intensive Care Unit (ICU) nurses. Patient self-report is subjective to the individual and suffers due to poor recall. Pain assessment by manual observation is limited by the number of administrations per day and staff workload. Previous studies showed the feasibility of automatic pain assessment by detecting Facial Action Units (AUs). Pain is observed to be associated with certain facial action units (AUs). This method of pain assessment can overcome the pitfalls of present-day pain assessment techniques. All the previous studies are limited to controlled environment data. In this study, we evaluated the performance of OpenFace an open-source facial behavior analysis tool and AU R-CNN on the real-world ICU data. Presence of assisted breathing devices, variable lighting of ICUs, patient orientation with respect to camera significantly affected the performance of the models, although these showed the state-of-the-art results in facial behavior analysis tasks. In this study, we show the need for automated pain assessment system which is trained on real-world ICU data for clinically acceptable pain assessment system.
Intelligent Critical Care Center, University of Florida(佛罗里达大学智能重症监护中心);University of Florida(佛罗里达大学)
Pain is a common occurrence among patients admitted to Intensive Care Units. Pain assessment in ICU patients still remains a challenge for clinicians and ICU staff, specifically in cases of non-verbal sedated, mechanically ventilated, and intubated patients. Current manual observation-based pain assessment tools are limited by the frequency of pain observations administered and are subjective to the observer. Facial behavior is a major component in observation-based tools. Furthermore, previous literature shows the feasibility of painful facial expression detection using facial action units (A…
Pain in 3D: Generating Controllable Synthetic Faces for Automated Pain Assessment
Vector Institute(向量研究所);KITE Research Institute(KITE研究机构);University Health Network(大学健康网络);Department of Computer Science, University of Toronto(多伦多大学计算机科学系);Department of Medical Imaging, University of Toronto(多伦多大学医学成像系);Institute of Biomedical Engineering, University of Toronto(多伦多大学生物医学工程研究所)
GraphAU-Pain: Graph-based Action Unit Representation for Pain Intensity Estimation
Department of Computer Science, University of Cambridge, Cambridge, UK(剑桥大学计算机科学系);Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland(奥卢大学机器视觉与信号分析中心)
PainSeeker: An Automated Method for Assessing Pain in Rats Through Facial Expressions
Affiliated Stomatological Hospital, Nanjing Medical University(南京医科大学附属口腔医院);School of Biological Science and Medical Engineering, Southeast University(东南大学生物科学与医学工程学院)
We propose a novel method for automatic pain intensity estimation from facial images based on the framework of kernel Conditional Ordinal Random Fields (KCORF). We extend this framework to account for heteroscedasticity on the output labels(i.e., pain intensity scores) and introduce a novel dynamic features, dynamic ranks, that impose temporal ordinal constraints on the static ranks (i.e., intensity scores). Our experimental results show that the proposed approach outperforms state-of-the art methods for sequence classification with ordinal data and other ordinal regression models. The approac…
FX Palo Alto Laboratory(FX帕洛阿尔托实验室);KK Women’s and Children’s Hospital(KK妇女儿童医院);Duke-NUS Medical School(杜克-新加坡国立大学医学院)
Managing post-surgical pain is critical for successful surgical outcomes. One of the challenges of pain management is accurately assessing the pain level of patients. Self-reported numeric pain ratings are limited because they are subjective, can be affected by mood, and can influence the patient's perception of pain when making comparisons. In this paper, we introduce an approach that analyzes 2D and 3D facial keypoints of post-surgical patients to estimate their pain intensity level. Our approach leverages the previously unexplored capabilities of a smartphone to capture a dense 3D represent…
In this paper, we present a process to investigate the effects of transfer learning for automatic facial expression recognition from emotions to pain. To this end, we first train a VGG16 convolutional neural network to automatically discern between eight categorical emotions. We then fine-tune successively larger parts of this network to learn suitable representations for the task of automatic pain recognition. Subsequently, we apply those fine-tuned representations again to the original task of emotion recognition to further investigate the differences in performance between the models. In th…
School of Computer Science, The University of Nottingham(诺丁汉大学计算机学院);University College London(伦敦大学学院);Department of Computer Science, University of East Anglia(东英吉利大学计算机科学系);Department of Electronic and Computer Engineering, Brunel University London(布鲁内尔大学伦敦电子与计算机工程系);Department of Computer Science, University of Oxford(牛津大学计算机科学系)
The EmoPain 2020 Challenge is the first international competition aimed at creating a uniform platform for the comparison of machine learning and multimedia processing methods of automatic chronic pain assessment from human expressive behaviour, and also the identification of pain-related behaviours. The objective of the challenge is to promote research in the development of assistive technologies that help improve the quality of life for people with chronic pain via real-time monitoring and feedback to help manage their condition and remain physically active. The challenge also aims to encour…
University of Illinois at Chicago(伊利诺伊大学芝加哥分校);University of Florida(佛罗里达大学)
Facial pain expression is an important modality for assessing pain, especially when the patient's verbal ability to communicate is impaired. The facial muscle-based action units (AUs), which are defined by the Facial Action Coding System (FACS), have been widely studied and are highly reliable as a method for detecting facial expressions (FE) including valid detection of pain. Unfortunately, FACS coding by humans is a very time-consuming task that makes its clinical use prohibitive. Significant progress on automated facial expression recognition (AFER) has led to its numerous successful applic…
Facial landmark detection plays an important role for the similarity analysis in artworks to compare portraits of the same or similar artists. With facial landmarks, portraits of different genres, such as paintings and prints, can be automatically aligned using control-point-based image registration. We propose a deep-learning-based method for facial landmark detection in high-resolution images of paintings and prints. It divides the task into a global network for coarse landmark prediction and multiple region networks for precise landmark refinement in regions of the eyes, nose, and mouth tha…
Automated facial recognition system using deep learning for pain assessment in adults with cerebral palsy
Balearic ASPACE Foundation(巴利阿里ASPACE基金会);University of the Balearic Islands(巴利阿里群岛大学);Research Institute on Health Sciences (IUNICS)(健康科学研究所 (IUNICS));Health Research Institute of the Balearic Islands (IdISBa)(巴利阿里群岛健康研究所 (IdISBa));Federal University of ABC(ABC联邦大学)
Sheep Facial Pain Assessment Under Weighted Graph Neural Networks
CISTER Research Center, Porto, Portugal(CISTER研究中心,葡萄牙 Porto);Faculty of Engineering University of Porto, Portugal(葡萄牙 Porto大学工程学院);Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000, Lille, France(法国 Lille 大学,CNRS,Centrale Lille,UMR 9189 CRIStAL);IMT Nord Europe, Institut Mines-Télécom, Univ. Lille, Centre for Digital Systems, F-59000 , Lille, France(法国 IMT Nord Europe,Institut Mines-Télécom,Univ. Lille,数字系统中心)
Pain is a personal, subjective experience that is commonly evaluated through visual analog scales (VAS). While this is often convenient and useful, automatic pain detection systems can reduce pain score acquisition efforts in large-scale studies by estimating it directly from the participants' facial expressions. In this paper, we propose a novel two-stage learning approach for VAS estimation: first, our algorithm employs Recurrent Neural Networks (RNNs) to automatically estimate Prkachin and Solomon Pain Intensity (PSPI) levels from face images. The estimated scores are then fed into the pers…
Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS
Computational Biomedicine Laboratory of the Foundation for Research and Technology (FORTH)(希腊国家研究与技术基金会计算生物医学实验室);Department of Electrical & Computer Engineering Hellenic Mediterranean University(希腊地中海大学电气与计算机工程系)
SynPAIN: A Synthetic Dataset of Pain and Non-Pain Facial Expressions
KITE Research Institute, Toronto Rehabilitation Institute, University Health Network(KITE研究 institute、多伦多康复研究所、大学健康网络);Department of Computer Science, University of Toronto(多伦多大学计算机科学系);Institute of Biomedical Engineering, University of Toronto(多伦多大学生物医学工程研究所);Vector Institute(向量研究所);Department of Occupational Science and Occupational Therapy, University of Toronto(多伦多大学职业科学与职业治疗系);University of Regina(Regina 多伦多大学)
Computer Science and Artificial Intelligence Laboratory, MIT(麻省理工学院计算机科学与人工智能实验室);Department of Electrical Engineering and Computer Science, MIT(麻省理工学院电气工程与计算机科学系);Media Lab, MIT(麻省理工学院媒体实验室)
Previous research on automatic pain estimation from facial expressions has focused primarily on "one-size-fits-all" metrics (such as PSPI). In this work, we focus on directly estimating each individual's self-reported visual-analog scale (VAS) pain metric, as this is considered the gold standard for pain measurement. The VAS pain score is highly subjective and context-dependent, and its range can vary significantly among different persons. To tackle these issues, we propose a novel two-stage personalized model, named DeepFaceLIFT, for automatic estimation of VAS. This model is based on (1) Neu…
Automatic pain intensity assessment has a high value in disease diagnosis applications. Inspired by the fact that many diseases and brain disorders can interrupt normal facial expression formation, we aim to develop a computational model for automatic pain intensity assessment from spontaneous and micro facial variations. For this purpose, we propose a 3D deep architecture for dynamic facial video representation. The proposed model is built by stacking several convolutional modules where each module encompasses a 3D convolution kernel with a fixed temporal depth, several parallel 3D convolutio…
Univ. Lille(里尔大学);CNRS(法国国家科学研究中心);Centrale Lille(里尔中央理工学院);IMT Lille Douai(里尔杜埃国立高等矿业电信学院);University of Florence(佛罗伦萨大学);Carnegie Mellon University(卡内基梅隆大学)
We propose an automatic method for pain intensity measurement from video. For each video, pain intensity was measured using the dynamics of facial movement using 66 facial points. Gram matrices formulation was used for facial points trajectory representations on the Riemannian manifold of symmetric positive semi-definite matrices of fixed rank. Curve fitting and temporal alignment were then used to smooth the extracted trajectories. A Support Vector Regression model was then trained to encode the extracted trajectories into ten pain intensity levels consistent with the Visual Analogue Scale fo…
3D Weighted Geometric Graph Neural Networks for Sheep Facial Pain Assessment
CISTER Research Center(CISTER研究中心);Faculty of Engineering University of Porto(波尔图大学工程学院);Univ. Lille(里尔大学);CNRS(法国国家科学研究中心);Centrale Lille(里尔中央理工学院);IMT Nord Europe(北欧电信学院);Institut Mines-Télécom(法国国立矿业与电信学院);Centre for Digital Systems(数字系统中心)
Graduate School of Information Science and Engineering, Ritsumeikan University, Japan(立命馆大学信息科学与工程研究生院);College of Information Science and Engineering, Ritsumeikan University, Japan(立命馆大学信息科学与工程学院);Faculty of Data Science, Shiga University, Japan(滋贺大学数据科学学院)
Currently, many critical care indices are repetitively assessed and recorded by overburdened nurses, e.g. physical function or facial pain expressions of nonverbal patients. In addition, many essential information on patients and their environment are not captured at all, or are captured in a non-granular manner, e.g. sleep disturbance factors such as bright light, loud background noise, or excessive visitations. In this pilot study, we examined the feasibility of using pervasive sensing technology and artificial intelligence for autonomous and granular monitoring of critically ill patients an…
University of California San Diego(加州大学圣地亚哥分校);Indian Institute of Technology Kanpur(印度理工学院坎普尔分校);MPI for Informatics(马克斯·普朗克信息学研究所);Emotient(Emotient公司)
We study the problem of facial analysis in videos. We propose a novel weakly supervised learning method that models the video event (expression, pain etc.) as a sequence of automatically mined, discriminative sub-events (eg. onset and offset phase for smile, brow lower and cheek raise for pain). The proposed model is inspired by the recent works on Multiple Instance Learning and latent SVM/HCRF- it extends such frameworks to model the ordinal or temporal aspect in the videos, approximately. We obtain consistent improvements over relevant competitive baselines on four challenging and publicly a…
KITE, Toronto Rehabilitation Institute, University Health Network(大学健康网络多伦多康复研究所KITE);University of Toronto(多伦多大学);University of Northern British Columbia(北不列颠哥伦比亚大学);University of Regina(里贾纳大学);Centre on Aging and Health, University of Regina(里贾纳大学衰老与健康中心)
Although pain is frequent in old age, older adults are often undertreated for pain. This is especially the case for long-term care residents with moderate to severe dementia who cannot report their pain because of cognitive impairments that accompany dementia. Nursing staff acknowledge the challenges of effectively recognizing and managing pain in long-term care facilities due to lack of human resources and, sometimes, expertise to use validated pain assessment approaches on a regular basis. Vision-based ambient monitoring will allow for frequent automated assessments so care staff could be au…
This paper proposes a new approach to model the temporal dynamics of a sequence of facial expressions. To this purpose, a sequence of Face Image Descriptors (FID) is regarded as the output of a Linear Time Invariant (LTI) system. The temporal dynamics of such sequence of descriptors are represented by means of a Hankel matrix. The paper presents different strategies to compute dynamics-based representation of a sequence of FID, and reports classification accuracy values of the proposed representations within different standard classification frameworks. The representations have been validated …
The quantification of visual affect data (e.g. face images) is essential to build and monitor automated affect modeling systems efficiently. Considering this, this work proposes quantified facial Temporal-expressiveness Dynamics (TED) to quantify the expressiveness of human faces. The proposed algorithm leverages multimodal facial features by incorporating static and dynamic information to enable accurate measurements of facial expressiveness. We show that TED can be used for high-level tasks such as summarization of unstructured visual data, and expectation from and interpretation of automate…
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford(牛津大学工程科学系生物医学工程研究所);Department of Paediatrics, University of Oxford(牛津大学儿科学系);Universidad San Francisco de Quito USFQ, Colegio de Ciencias Biologicas y Ambientales(基多圣弗朗西斯科大学生物与环境科学学院)
University of Oulu(奥卢大学);École de Technologie Supérieure(高等技术学院);VTT Technical Research Centre of Finland Ltd(芬兰VTT技术研究中心有限公司)
Video-based analysis of facial expressions has been increasingly applied to infer health states of individuals, such as depression and pain. Among the existing approaches, deep learning models composed of structures for multiscale spatiotemporal processing have shown strong potential for encoding facial dynamics. However, such models have high computational complexity, making for a difficult deployment of these solutions. To address this issue, we introduce a new technique to decompose the extraction of multiscale spatiotemporal features. Particularly, a building block structure called Decompo…
Pain Analysis using Adaptive Hierarchical Spatiotemporal Dynamic Imaging
IEMN - DOAE UMR CNRS 8520;Université Polytechnique Hauts-de-France(上法兰西理工大学);École de technologie supérieure(高等技术学院);Université du Québec Montreal(魁北克大学蒙特利尔分校);Sorbonne Center for Artificial Intelligence(索邦人工智能中心)
Intelligent Sight and Sound: A Chronic Cancer Pain Dataset
Booz Allen Hamilton(博思艾伦汉密尔顿公司);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Center for Cancer Research, National Cancer Institute, National Institutes of Health(美国国立卫生研究院国家癌症研究所癌症研究中心)
This paper presents the first investigation into the use of fully automated deep learning framework for assessing neonatal postoperative pain. It specifically investigates the use of Bilinear Convolutional Neural Network (B-CNN) to extract facial features during different levels of postoperative pain followed by modeling the temporal pattern using Recurrent Neural Network (RNN). Although acute and postoperative pain have some common characteristics (e.g., visual action units), postoperative pain has a different dynamic, and it evolves in a unique pattern over time. Our experimental results ind…
KITE Research Institute(KITE研究院);Toronto Rehabilitation Institute - University Health Network(多伦多康复研究所-大学健康网络);University of Toronto(多伦多大学);Peter Munk Cardiac Centre, University Health Network(大学健康网络彼得·蒙克心脏中心)
Pain monitoring is essential to the quality of care for patients undergoing a medical procedure with sedation. An automated mechanism for detecting pain could improve sedation dose titration. Previous studies on facial pain detection have shown the viability of computer vision methods in detecting pain in unoccluded faces. However, the faces of patients undergoing procedures are often partially occluded by medical devices and face masks. A previous preliminary study on pain detection on artificially occluded faces has shown a feasible approach to detect pain from a narrow band around the eyes.…
Tel Aviv University(特拉维夫大学);Geha Mental Health Center(格哈心理健康中心)
Automatic recognition of facial gestures is becoming increasingly important as real world AI agents become a reality. In this paper, we present an automated system that recognizes facial gestures by capturing local changes and encoding the motion into a histogram of frequencies. We evaluate the proposed method by demonstrating its effectiveness on spontaneous face action benchmarks: the FEEDTUM dataset, the Pain dataset and the HMDB51 dataset. The results show that, compared to known methods, the new encoding methods significantly improve the recognition accuracy and the robustness of analysis…
Pain assessment through observational pain scales is necessary for special categories of patients such as neonates, patients with dementia, critically ill patients, etc. The recently introduced Prkachin-Solomon score allows pain assessment directly from facial images opening the path for multiple assistive applications. In this paper, we introduce the Histograms of Topographical (HoT) features, which are a generalization of the topographical primal sketch, for the description of the face parts contributing to the mentioned score. We propose a semi-supervised, clustering oriented self--taught l…
Gender Fairness of Machine Learning Algorithms for Pain Detection
Department of Computer Science and Technology, University of Cambridge(计算机科学与技术系,剑桥大学);Harvard University(哈佛大学);Center for Machine Vision and Signal Analysis, University of Oulu(图像与信号分析中心,奥卢大学)
University of South Florida(南佛罗里达大学);National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);USF Health(南佛罗里达大学健康学院)
Neonatal pain assessment in clinical environments is challenging as it is discontinuous and biased. Facial/body occlusion can occur in such settings due to clinical condition, developmental delays, prone position, or other external factors. In such cases, crying sound can be used to effectively assess neonatal pain. In this paper, we investigate the use of a novel CNN architecture (N-CNN) along with other CNN architectures (VGG16 and ResNet50) for assessing pain from crying sounds of neonates. The experimental results demonstrate that using our novel N-CNN for assessing pain from the sounds of…
Limited labeled data are available for the research of estimating facial expression intensities. For instance, the ability to train deep networks for automated pain assessment is limited by small datasets with labels of patient-reported pain intensities. Fortunately, fine-tuning from a data-extensive pre-trained domain, such as face verification, can alleviate this problem. In this paper, we propose a network that fine-tunes a state-of-the-art face verification network using a regularized regression loss and additional data with expression labels. In this way, the expression intensity regressi…
Pain is a serious and costly issue globally, but to be treated, it must first be detected. Vision transformers are a top-performing architecture in computer vision, with little research on their use for pain detection. In this paper, we propose the first fully-attentive automated pain detection pipeline that achieves state-of-the-art performance on binary pain detection from facial expressions. The model is trained on the UNBC-McMaster dataset, after faces are 3D-registered and rotated to the canonical frontal view. In our experiments we identify important areas of the hyperparameter space and…
FG 2025 TrustFAA: the First Workshop on Towards Trustworthy Facial Affect Analysis: Advancing Insights of Fairness, Explainability, and Safety (TrustFAA)
University of Cambridge(剑桥大学);University of Oulu(奥卢大学);National University of Singapore(新加坡国立大学)
Neonatal Face and Facial Landmark Detection from Video Recordings
Monash University(莫纳什大学);University British Columbia(不列颠哥伦比亚大学);Monash Newborn(莫纳什新生儿中心);Monash Institute of Medical Engineering(莫纳什医学工程研究所);Monash Data Future Institute(莫纳什数据未来研究所)
Georgian Partners Inc(Georgian Partners公司);University of Toronto(多伦多大学);Toronto Rehabilitation Institute, University Health Network(大学健康网络多伦多康复研究所);University of Manitoba(曼尼托巴大学);University of Regina(里贾纳大学);University of Northern British Columbia(北不列颠哥伦比亚大学);Vector Institute for Artificial Intelligence(Vector人工智能研究所)
Accurate facial expression analysis is an essential step in various clinical applications that involve physical and mental health assessments of older adults (e.g. diagnosis of pain or depression). Although remarkable progress has been achieved toward developing robust facial landmark detection methods, state-of-the-art methods still face many challenges when encountering uncontrolled environments, different ranges of facial expressions, and different demographics of the population. A recent study has revealed that the health status of individuals can also affect the performance of facial land…
Unsupervised learning of Data-driven Facial Expression Coding System (DFECS) using keypoint tracking
Indian Institute of Technology Delhi(印度理工学院德里分校);Amar Nath and Shashi Khosla School of Information Technology, Indian Institute of Technology Delhi(印度理工学院德里分校阿马尔·纳特和夏希·霍斯拉信息技术学院);National Resource Centre for Value Education in Engineering, Indian Institute of Technology Delhi(印度理工学院德里分校工程价值教育国家资源中心)
Fraunhofer Institute for Integrated Circuits IIS(弗劳恩霍夫集成电路研究所)
Isolated facial movements, so-called Action Units, can describe combined emotions or physical states such as pain. As datasets are limited and mostly imbalanced, we present an approach incorporating a multi-label class balancing algorithm. This submission is subject to the Action Unit detection task of the Affective Behavior Analysis in-the-wild (ABAW) challenge at the IEEE Conference on Face and Gesture Recognition 2020.
SRI International(斯坦福国际研究院);University of California San Diego(加利福尼亚大学圣迭戈分校);Indian Institute of Technology Kanpur(印度理工学院坎普尔分校);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)
We study the problem of video classification for facial analysis and human action recognition. We propose a novel weakly supervised learning method that models the video as a sequence of automatically mined, discriminative sub-events (eg. onset and offset phase for "smile", running and jumping for "highjump"). The proposed model is inspired by the recent works on Multiple Instance Learning and latent SVM/HCRF -- it extends such frameworks to model the ordinal aspect in the videos, approximately. We obtain consistent improvements over relevant competitive baselines on four challenging and publi…
Horse Eye Blink Detection and Classification for Equine Affective State Assessment
Visual Analysis and Perception Lab, Aalborg University(视觉分析与感知实验室,奥尔堡大学);Department of Animal Biosciences, Swedish University of Agricultural Sciences(动物生物科学系,瑞典农业科学大学)
Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data
LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada(LIVIA,系统工程系,蒙特利尔大学工程学院,加拿大);LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada(LIVIA,软件与信息技术工程系,蒙特利尔大学工程学院,加拿大);Dept. of Health, Kinesiology & Applied Physiology, Concordia University, Montreal, Canada(健康、运动科学与应用生理学系,康科迪亚大学,蒙特利尔,加拿大)
Fraunhofer Institute for Integrated Circuits IIS(弗劳恩霍夫集成电路研究所)
Balancing methods for single-label data cannot be applied to multi-label problems as they would also resample the samples with high occurrences. We propose to reformulate this problem as an optimization problem in order to balance multi-label data. We apply this balancing algorithm to training datasets for detecting isolated facial movements, so-called Action Units. Several Action Units can describe combined emotions or physical states such as pain. As datasets in this area are limited and mostly imbalanced, we show how optimized balancing and then augmentation can improve Action Unit detectio…
Faithful Action-unit Causal Reasoning for Counterfactually Faithful Emotion Explanations
Faculty of CSE, Ho Chi Minh City University of Technology (HCMUT), VNUHCM(胡志明市理工大学计算机科学与工程学院,越南国家大学胡志明市分校);Dept. of AI, FPT University(FPT大学人工智能系);Faculty of DSAI, College of Technology, National Economic University(国民经济大学技术学院数据科学与人工智能系);Dept. of AI Convergence, Chonnam National University(全南大学人工智能融合系)
InfantFace: Detecting infant faces in neonatal clinical environments
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford(牛津大学生物医学工程研究所、工程科学系);Department of Paediatrics, University of Oxford(牛津大学儿科系);Universidad San Francisco de Quito USFQ, Colegio de Ciencias Biologicas y Ambientales(奎托大学圣弗朗西斯科德奎托大学,生物科学与环境学院)