This paper presents a comparative evaluation of methods for remote heart rate estimation using face videos, i.e., given a video sequence of the face as input, methods to process it to obtain a robust estimation of the subjects heart rate at each moment. Four alternatives from the literature are tested, three based in hand crafted approaches and one based on deep learning. The methods are compared using RGB videos from the COHFACE database. Experiments show that the learning-based method achieves much better accuracy than the hand crafted ones. The low error rate achieved by the learning based model makes possible its application in real scenarios, e.g. in medical or sports environments.
Extracting the instantaneous heart rate (iHR) from face videos has been well studied in recent years. It is well known that changes in skin color due to blood flow can be captured using conventional cameras. One of the main limitations of methods that rely on this principle is the need of an illumination source. Moreover, they have to be able to operate under different light conditions. One way to avoid these constraints is using infrared cameras, allowing the monitoring of iHR under low light conditions. In this work, we present a simple, principled signal extraction method that recovers the …
TU Delft（代尔夫特理工大学）；VU Amsterdam（阿姆斯特丹自由大学）；Radboud UMC（拉德堡德大学医学中心）
Estimating heart rate from video allows non-contact health monitoring with applications in patient care, human interaction, and sports. Existing work can robustly measure heart rate under some degree of motion by face tracking. However, this is not always possible in unconstrained settings, as the face might be occluded or even outside the camera. Here, we present IntensePhysio: a challenging video heart rate estimation dataset with realistic face occlusions, severe subject motion, and ample heart rate variation. To ensure heart rate variation in a realistic setting we record each subject for …
IIT Indore（印度理工学院印多尔分校）；TCS Research and Innovation（塔塔咨询服务公司研究与创新部）
A non-invasive yet inexpensive method for heart rate (HR) monitoring is of great importance in many real-world applications including healthcare, psychology understanding, affective computing and biometrics. Face videos are currently utilized for such HR monitoring, but unfortunately this can lead to errors due to the noise introduced by facial expressions, out-of-plane movements, camera parameters (like focus change) and environmental factors. We alleviate these issues by proposing a novel face video based HR monitoring method MOMBAT, that is, MOnitoring using Modeling and BAyesian Tracking. …
In recent years, research about monitoring vital signs by smartphones grows significantly. There are some special sensors like Electrocardiogram (ECG) and Photoplethysmographic (PPG) to detect heart rate (HR) and respiration rate (RR). Smartphone cameras also can measure HR by detecting and processing imaging Photoplethysmographic (iPPG) signals from the video of a user's face. Indeed, the variation in the intensity of the green channel can be measured by the iPPG signals of the video. This study aimed to provide a method to extract heart rate and respiration rate using the video of individual…
As the GAN-based face image and video generation techniques, widely known as DeepFakes, have become more and more matured and realistic, there comes a pressing and urgent demand for effective DeepFakes detectors. Motivated by the fact that remote visual photoplethysmography (PPG) is made possible by monitoring the minuscule periodic changes of skin color due to blood pumping through the face, we conjecture that normal heartbeat rhythms found in the real face videos will be disrupted or even entirely broken in a DeepFake video, making it a potentially powerful indicator for DeepFake detection. …
Dalio Institute of Cardiovascular Imaging, Weill Cornell Medical College（威尔康奈尔医学院达利奥心血管影像研究所）
Recently, a successful pose estimation algorithm, called Cascade Pose Regression (CPR), was proposed in the literature. Trained over Pose Index Feature, CPR is a regressor ensemble that is similar to Boosting. In this paper we show how CPR can be represented as a Neural Network. Specifically, we adopt a Graph Transformer Network (GTN) representation and accordingly train CPR with Back Propagation (BP) that permits globally tuning. In contrast, previous CPR literature only took a layer wise training without any post fine tuning. We empirically show that global training with BP outperforms layer…
In this paper we present a comparative study on fusion of visual and thermal images using different wavelet transformations. Here, coefficients of discrete wavelet transforms from both visual and thermal images are computed separately and combined. Next, inverse discrete wavelet transformation is taken in order to obtain fused face image. Both Haar and Daubechies (db2) wavelet transforms have been used to compare recognition results. For experiments IRIS Thermal/Visual Face Database was used. Experimental results using Haar and Daubechies wavelets show that the performance of the approach pres…
Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM））；University of Oulu（奥卢大学）；Iran University of Science and Technology（伊朗科学技术大学）
This paper presents a novel method for remote heart rate (HR) estimation. Recent studies have proved that blood pumping by the heart is highly correlated to the intense color of face pixels, and surprisingly can be utilized for remote HR estimation. Researchers successfully proposed several methods for this task, but making it work in realistic situations is still a challenging problem in computer vision community. Furthermore, learning to solve such a complex task on a dataset with very limited annotated samples is not reasonable. Consequently, researchers do not prefer to use the deep learni…
SENAI CIMATEC University Center（SENAI CIMATEC大学中心）
Estimating heart rate is important for monitoring users in various situations. Estimates based on facial videos are increasingly being researched because it makes it possible to monitor cardiac information in a non-invasive way and because the devices are simpler, requiring only cameras that capture the user's face. From these videos of the user's face, machine learning is able to estimate heart rate. This study investigates the benefits and challenges of using machine learning models to estimate heart rate from facial videos, through patents, datasets, and articles review. We searched Derwent…
Facial recognition is fundamental for a wide variety of security systems operating in real-time applications. In video surveillance based face recognition, face images are typically captured over multiple frames in uncontrolled conditions; where head pose, illumination, shadowing, motion blur and focus change over the sequence. We can generalize that the three fundamental operations involved in the facial recognition tasks: face detection, face alignment and face recognition. This study presents comparative benchmark tables for the state-of-art face recognition methods by testing them with sam…
The World Health Organization (WHO) has recommended wearing face masks as one of the most effective measures to prevent COVID-19 transmission. In many countries, it is now mandatory to wear face masks, specially in public places. Since manual monitoring of face masks is often infeasible in the middle of the crowd, automatic detection can be beneficial. To facilitate that, we explored a number of deep learning models (i.e., VGG1, VGG19, ResNet50) for face-mask detection and evaluated them on two benchmark datasets. We also evaluated transfer learning (i.e., VGG19, ResNet50 pre-trained on ImageN…
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation…
Eötvös Loránd University（厄特沃什·罗兰大学）；Mutah University（穆塔大学）；Gazi University（加齐大学）；BUET（孟加拉国工程技术大学）
Facial image retrieval is a challenging task since faces have many similar features (areas), which makes it difficult for the retrieval systems to distinguish faces of different people. With the advent of deep learning, deep networks are often applied to extract powerful features that are used in many areas of computer vision. This paper investigates the application of different deep learning models for face image retrieval, namely, Alexlayer6, Alexlayer7, VGG16layer6, VGG16layer7, VGG19layer6, and VGG19layer7, with two types of dictionary learning techniques, namely $K$-means and $K$-SVD. We …
In modern times, face recognition has become one of the key aspects of computer vision. There are at least two reasons for this trend; the first is the commercial and law enforcement applications, and the second is the availability of feasible technologies after years of research. Due to the very nature of the problem, computer scientists, neuro-scientists and psychologists all share a keen interest in this field. In plain words, it is a computer application for automatically identifying a person from a still image or video frame. One of the ways to accomplish this is by comparing selected fea…
Thermal infra-red (IR) images focus on changes of temperature distribution on facial muscles and blood vessels. These temperature changes can be regarded as texture features of images. A comparative study of face recognition methods working in thermal spectrum is carried out in this paper. In these study two local-matching methods based on Haar wavelet transform and Local Binary Pattern (LBP) are analyzed. Wavelet transform is a good tool to analyze multi-scale, multi-direction changes of texture. Local binary patterns (LBP) are a type of feature used for classification in computer vision. Fir…
Estimation of the Heart rate from the facial video has a number of applications in the medical and fitness industries. Additionally, it has become useful in the field of gaming as well. Several approaches have been proposed to seamlessly obtain the Heart rate from the facial video, but these approaches have had issues in dealing with motion and illumination artifacts. In this work, we propose a reliable HR estimation framework using the spectral reflectance of the user, which makes it robust to motion and illumination disturbances. We employ deep learning-based frameworks such as Faster RCNNs …
Beam AI；University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）
Stress is considered to be the epidemic of the 21st-century. Yet, mobile apps cannot directly evaluate the impact of their content and services on user stress. We introduce the Beam AI SDK to address this issue. Using our SDK, apps can monitor user stress through the selfie camera in real-time. Our technology extracts the user's pulse wave by analyzing subtle color variations across the skin regions of the user's face. The user's pulse wave is then used to determine stress (according to the Baevsky Stress Index), heart rate, and heart rate variability. We evaluate our technology on the UBFC da…
VideoPulse: Neonatal heart rate and peripheral capillary oxygen saturation (SpO2) estimation from contact free video
Department of Electronic and Telecommunication Engineering, University of Moratuwa（电子与电信工程系，穆塔瓦大学）；School of Computer Science, Faculty of Engineering, The University of Sydney（计算机科学学院，悉尼大学）；Department of Paediatrics, Faculty of Medicine, University of Colombo（儿科系，科伦坡大学）
Face De-identification: State-of-the-art Methods and Comparative Studies
The Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；University of Technology Sydney（悉尼科技大学）；Data61, CSIRO（联邦科学与工业研究组织Data61研究院）；University of Missouri-Kansas City（密苏里大学堪萨斯城分校）
Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形学研究所IGD）；Instituto Tecnológico Metropolitano - ITM（大都会理工学院（ITM））；Saarland University（萨尔大学）
With systems for acquiring 3D surface data being evermore commonplace, it has become important to reliably extract specific shapes from the acquired data. In the presence of noise and occlusions, this can be done through the use of statistical shape models, which are learned from databases of clean examples of the shape in question. In this paper, we review, analyze and compare different statistical models: from those that analyze the variation in geometry globally to those that analyze the variation in geometry locally. We first review how different types of models have been used in the liter…
A Comparative Study on Synthetic Facial Data Generation Techniques for Face Recognition
Department of Informatics, Federal University of Paraná（帕拉那联邦大学信息学院）；IFMT, Pontes e Lacerda（Pontes e Lacerda 工业技术学院）；UFMG（伯南布哥大学）；Unico IDTech（Unico IDTech 公司）
Deep Network Pruning: A Comparative Study on CNNs in Face Recognition
School of Information Science, Computer and Electrical Engineering, Halmstad University（信息科学、计算机与电气工程学院，哈姆斯塔德大学）；Computer Graphics and Vision and AI Group, University of Balearic Islands（计算机图形学与视觉与人工智能小组，巴利阿里大学）
Chaos in Motion: Unveiling Robustness in Remote Heart Rate Measurement through Brain-Inspired Skin Tracking
School of Information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）；School of Electronics and Information Engineering, Lanzhou Jiaotong University（兰州交通大学电子与信息工程学院）；The First Hospital of Lanzhou University（兰州大学第一医院）；School of Artificial Intelligence, Nanjing University of Information Science and Technology（南京信息工程大学人工智能学院）
Institute of Robotics Innopolis University（Innopolis大学机器人研究所）
Human heartbeat can be measured using several different ways appropriately based on the patient condition which includes contact base such as measured by using instruments and non-contact base such as computer vision assisted techniques. Non-contact based approached are getting popular due to those techniques are capable of mitigating some of the limitations of contact-based techniques especially in clinical section. However, existing vision guided approaches are not able to prove high accurate result due to various reason such as the property of camera, illumination changes, skin tones in fac…
In this paper, we present a face fairness framework for 3D meshes that preserves the regular shape of faces and is applicable to a variety of 3D mesh restoration tasks. Specifically, we present a number of desirable properties for any mesh restoration method and show that our framework satisfies them. We then apply our framework to two different tasks --- mesh-denoising and mesh-refinement, and present comparative results for these two tasks showing improvement over other relevant methods in the literature.
We propose an end-to-end framework to measure people's vital signs including Heart Rate (HR), Heart Rate Variability (HRV), Oxygen Saturation (SpO2) and Blood Pressure (BP) based on the rPPG methodology from the video of a user's face captured with a smartphone camera. We extract face landmarks with a deep learning-based neural network model in real-time. Multiple face patches also called Regions-of-Interest (RoIs) are extracted by using the predicted face landmarks. Several filters are applied to reduce the noise from the RoIs in the extracted cardiac signals called Blood Volume Pulse (BVP) s…
Beijing University of Posts and Telecommunications（北京邮电大学）
With the emergence of GAN, face forgery technologies have been heavily abused. Achieving accurate face forgery detection is imminent. Inspired by remote photoplethysmography (rPPG) that PPG signal corresponds to the periodic change of skin color caused by heartbeat in face videos, we observe that despite the inevitable loss of PPG signal during the forgery process, there is still a mixture of PPG signals in the forgery video with a unique rhythmic pattern depending on its generation method. Motivated by this key observation, we propose a framework for face forgery detection and categorization …
Universidad Autonoma de Madrid（马德里自治大学）；OMRON Corporation（欧姆龙公司）
In this paper we develop a robust for heart rate (HR) estimation method using face video for challenging scenarios with high variability sources such as head movement, illumination changes, vibration, blur, etc. Our method employs a quality measure Q to extract a remote Plethysmography (rPPG) signal as clean as possible from a specific face video segment. Our main motivation is developing robust technology for driver monitoring. Therefore, for our experiments we use a self-collected dataset consisting of Near Infrared (NIR) videos acquired with a camera mounted in the dashboard of a real movin…
Camera-based physiological monitoring, especially remote photoplethysmography (rPPG), is a promising tool for health diagnostics, and state-of-the-art pulse estimators have shown impressive performance on benchmark datasets. We argue that evaluations of modern solutions may be incomplete, as we uncover failure cases for videos without a live person, or in the presence of severe noise. We demonstrate that spatiotemporal deep learning models trained only with live samples "hallucinate" a genuine-shaped pulse on anomalous and noisy videos, which may have negative consequences when rPPG models are…
Beihang University（北京航空航天大学）；Zhejiang University of Technology（浙江工业大学）
With the improvement of sensor technology and significant algorithmic advances, the accuracy of remote heart rate monitoring technology has been significantly improved. Despite of the significant algorithmic advances, the performance of rPPG algorithm can degrade in the long-term, high-intensity continuous work occurred in evenings or insufficient light environments. One of the main challenges is that the lost facial details and low contrast cause the failure of detection and tracking. Also, insufficient lighting in video capturing hurts the quality of physiological signal. In this paper, we c…
Remote photoplethysmography (rPPG), a family of techniques for monitoring blood volume changes, may be especially useful for widespread contactless health monitoring using face video from consumer-grade visible-light cameras. The COVID-19 pandemic has caused the widespread use of protective face masks. We found that occlusions from cloth face masks increased the mean absolute error of heart rate estimation by more than 80\% when deploying methods designed on unmasked faces. We show that augmenting unmasked face videos by adding patterned synthetic face masks forces the model to attend to the p…
This paper explores a comparative study of both the linear and kernel implementations of three of the most popular Appearance-based Face Recognition projection classes, these being the methodologies of Principal Component Analysis, Linear Discriminant Analysis and Independent Component Analysis. The experimental procedure provides a platform of equal working conditions and examines the ten algorithms in the categories of expression, illumination, occlusion and temporal delay. The results are then evaluated based on a sequential combination of assessment tools that facilitate both intuitive and…
Halmstad University（哈尔姆斯塔德大学）；Norwegian University of Science and Technology（挪威科技大学）；Universidad Autonoma de Madrid（马德里自治大学）
The massive availability of cameras results in a wide variability of imaging conditions, producing large intra-class variations and a significant performance drop if heterogeneous images are compared for person recognition. However, as biometrics is deployed, it is common to replace damaged or obsolete hardware, or to exchange information between heterogeneous applications. Variations in spectral bands can also occur. For example, surveillance face images (typically acquired in the visible spectrum, VIS) may need to be compared against a legacy iris database (typically acquired in near-infrare…
Video-based physiological signal estimation has been limited primarily to predicting episodic scores in windowed intervals. While these intermittent values are useful, they provide an incomplete picture of patients' physiological status and may lead to late detection of critical conditions. We propose a video Transformer for estimating instantaneous heart rate and respiration rate from face videos. Physiological signals are typically confounded by alignment errors in space and time. To overcome this, we formulated the loss in the frequency domain. We evaluated the method on the large scale Vis…
The ability to reliably estimate physiological signals from video is a powerful tool in low-cost, pre-clinical health monitoring. In this work we propose a new approach to remote photoplethysmography (rPPG) - the measurement of blood volume changes from observations of a person's face or skin. Similar to current state-of-the-art methods for rPPG, we apply neural networks to learn deep representations with invariance to nuisance image variation. In contrast to such methods, we employ a fully self-supervised training approach, which has no reliance on expensive ground truth physiological trainin…
The area of face recognition is one of the most widely researched areas in the domain of computer vision and biometric. This is because, the non-intrusive nature of face biometric makes it comparatively more suitable for application in area of surveillance at public places such as airports. The application of primitive methods in face recognition could not give very satisfactory performance. However, with the advent of machine and deep learning methods and their application in face recognition, several major breakthroughs were obtained. The use of 2D Convolution Neural networks(2D CNN) in face…
Recent studies demonstrated that the average heart rate (HR) can be measured from facial videos based on non-contact remote photoplethysmography (rPPG). However for many medical applications (e.g., atrial fibrillation (AF) detection) knowing only the average HR is not sufficient, and measuring precise rPPG signals from face for heart rate variability (HRV) analysis is needed. Here we propose an rPPG measurement method, which is the first work to use deep spatio-temporal networks for reconstructing precise rPPG signals from raw facial videos. With the constraint of trend-consistency with ground…
University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Heart rate (HR) is an important physiological signal that reflects the physical and emotional activities of humans. Traditional HR measurements are mainly based on contact monitors, which are inconvenient and may cause discomfort for the subjects. Recently, methods have been proposed for remote HR estimation from face videos. However, most of the existing methods focus on well-controlled scenarios, their generalization ability into less-constrained scenarios are not known. At the same time, lacking large-scale databases has limited the use of deep representation learning methods in remote HR e…
With a surge in online medical advising remote monitoring of patient vitals is required. This can be facilitated with the Remote Photoplethysmography (rPPG) techniques that compute vital signs from facial videos. It involves processing video frames to obtain skin pixels, extracting the cardiac data from it and applying signal processing filters to extract the Blood Volume Pulse (BVP) signal. Different algorithms are applied to the BVP signal to estimate the various vital signs. We implemented a web application framework to measure a person's Heart Rate (HR), Heart Rate Variability (HRV), Oxyge…
