Facial Kinship Verification (FKV) aims at automatically determining whether two subjects have a kinship relation based on human faces. It has potential applications in finding missing children and social media analysis. Traditional FKV faces challenges as it is vulnerable to spoof attacks and raises privacy issues. In this paper, we explore for the first time the FKV with vital bio-signals, focusing on remote Photoplethysmography (rPPG). rPPG signals are extracted from facial videos, resulting in a one-dimensional signal that measures the changes in visible light reflection emitted to and detected from the skin caused by the heartbeat. Specifically, in this paper, we employed a straightforward one-dimensional Convolutional Neural Network (1DCNN) with a 1DCNN-Attention module and kinship contrastive loss to learn the kin similarity from rPPGs. The network takes multiple rPPG signals extracted from various facial Regions of Interest (ROIs) as inputs. Additionally, the 1DCNN attention module is designed to learn and capture the discriminative kin features from feature embeddings. Finally, we demonstrate the feasibility of rPPG to detect kinship with the experiment evaluation on the UvANEMO Smile Database from different kin relations.
Imperial College London（伦敦帝国理工学院）；Middlesex University London（伦敦米德塞克斯大学）
Computational facial models that capture properties of facial cues related to aging and kinship increasingly attract the attention of the research community, enabling the development of reliable methods for age progression, age estimation, age-invariant facial characterization, and kinship verification from visual data. In this paper, we review recent advances in modeling of facial aging and kinship. In particular, we provide an up-to date, complete list of available annotated datasets and an in-depth analysis of geometric, hand-crafted, and learned facial representations that are used for fac…
University of Southern California（南加州大学）；Samsung Research America（三星美国研究院）
Kinship verification aims to identify the kin relation between two given face images. It is a very challenging problem due to the lack of training data and facial similarity variations between kinship pairs. In this work, we build a novel appearance and shape based deep learning pipeline. First we adopt the knowledge learned from general face recognition network to learn general facial features. Afterwards, we learn kinship oriented appearance and shape features from kinship pairs and combine them for the final prediction. We have evaluated the model performance on a widely used popular benchm…
University of Biskra（比斯克拉大学）；Univ. Polytechnique Hauts-de-France（上法兰西理工大学）；University of Oulu（奥卢大学）
Automatic kinship verification from facial images is an emerging research topic in machine learning community. In this paper, we proposed an effective facial features extraction model based on multi-view deep features. Thus, we used four pre-trained deep learning models using eight features layers (FC6 and FC7 layers of each VGG-F, VGG-M, VGG-S and VGG-Face models) to train the proposed Multilinear Side-Information based Discriminant Analysis integrating Within Class Covariance Normalization (MSIDA+WCCN) method. Furthermore, we show that how can metric learning methods based on WCCN method int…
Centre de Développement des Technologies Avancées（先进技术发展中心）；University of Oulu（奥卢大学）；Ecole Nationale Supèrieure d’Informatique（国立高等信息学学院）；Northwestern Polytechnical University（西北工业大学）
Automatic kinship verification using facial images is a relatively new and challenging research problem in computer vision. It consists in automatically predicting whether two persons have a biological kin relation by examining their facial attributes. While most of the existing works extract shallow handcrafted features from still face images, we approach this problem from spatio-temporal point of view and explore the use of both shallow texture features and deep features for characterizing faces. Promising results, especially those of deep features, are obtained on the benchmark UvA-NEMO Smi…
Kinship verification is a long-standing research challenge in computer vision. The visual differences presented to the face have a significant effect on the recognition capabilities of the kinship systems. We argue that aggregating multiple visual knowledge can better describe the characteristics of the subject for precise kinship identification. Typically, the age-invariant features can represent more natural facial details. Such age-related transformations are essential for face recognition due to the biological effects of aging. However, the existing methods mainly focus on employing the si…
University of Oulu（奥卢大学）；Northwestern Polytechnical University（西北工业大学）；École de technologie supérieure（高等技术学院）
Visual kinship verification entails confirming whether or not two individuals in a given pair of images or videos share a hypothesized kin relation. As a generalized face verification task, visual kinship verification is particularly difficult with low-quality found Internet data. Due to uncontrolled variations in background, pose, facial expression, blur, illumination and occlusion, state-of-the-art methods fail to provide high level of recognition accuracy. As with many other visual recognition tasks, kinship verification may benefit from combining visual and audio signals. However, voice-ba…
Orientation-conditioned Facial Texture Mapping for Video-based Facial Remote Photoplethysmography Estimation
Australian National University（澳大利亚国立大学）；Data61, Commonwealth and Scientific Industrial Research Organization（联邦科学与工业研究组织Data61）；University of Turku（图尔库大学）
Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
In this paper, we investigate the problem of facial kinship verification by learning hierarchical reasoning graph networks. Conventional methods usually focus on learning discriminative features for each facial image of a paired sample and neglect how to fuse the obtained two facial image features and reason about the relations between them. To address this, we propose a Star-shaped Reasoning Graph Network (S-RGN). Our S-RGN first constructs a star-shaped graph where each surrounding node encodes the information of comparisons in a feature dimension and the central node is employed as the brid…
Ecole Nationale Supérieure d’Informatique Algiers（阿尔及尔国立高等计算机学院）
Facial Kinship Verification is the task of determining the degree of familial relationship between two facial images. It has recently gained a lot of interest in various applications spanning forensic science, social media, and demographic studies. In the past decade, deep learning-based approaches have emerged as a promising solution to this problem, achieving state-of-the-art performance. In this paper, we propose a novel method for solving kinship verification by using supervised contrastive learning, which trains the model to maximize the similarity between related individuals and minimize…
Faculty of Information Technology, University of Science, VNU-HCM（越南国家大学胡志明市分校科学大学信息技术学院）
Kinship verification and kinship retrieval are emerging tasks in computer vision. Kinship verification aims at determining whether two facial images are from related people or not, while kinship retrieval is the task of retrieving possible related facial images to a person from a gallery of images. They introduce unique challenges because of the hidden relations and features that carry inherent characteristics between the facial images. We employ 3 methods, FaceNet, Siamese VGG-Face, and a combination of FaceNet and VGG-Face models as feature extractors, to achieve the 9th standing for kinship…
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Beijing University of Posts and Telecommunications（北京邮电大学）
Kinship verification aims to find out whether there is a kin relation for a given pair of facial images. Kinship verification databases are born with unbalanced data. For a database with N positive kinship pairs, we naturally obtain N(N-1) negative pairs. How to fully utilize the limited positive pairs and mine discriminative information from sufficient negative samples for kinship verification remains an open issue. To address this problem, we propose a Discriminative Sample Meta-Mining (DSMM) approach in this paper. Unlike existing methods that usually construct a balanced dataset with fixed…
MVRD-Bench: Multi-View Learning and Benchmarking for Dynamic Remote Photoplethysmography under Occlusion
Nanjing University of Information Science and Technology（南京信息工程大学）；University of Oulu（奥卢大学）；Xi’an Jiaotong University（西安交通大学）；Zhejiang University（浙江大学）
We present a lightweight neural model for remote heart rate estimation focused on the efficient spatio-temporal learning of facial photoplethysmography (PPG) based on i) modelling of PPG dynamics by combinations of multiple convolutional derivatives, and ii) increased flexibility of the model to learn possible offsets between the facial video PPG and the ground truth. PPG dynamics are modelled by a Temporal Derivative Module (TDM) constructed by the incremental aggregation of multiple convolutional derivatives, emulating a Taylor series expansion up to the desired order. Robustness to ground t…
Remote photoplethysmography (rPPG) enables non-contact heart rate (HR) estimation from facial videos which gives significant convenience compared with traditional contact-based measurements. In the real-world long-term health monitoring scenario, the distance of the participants and their head movements usually vary by time, resulting in the inaccurate rPPG measurement due to the varying face resolution and complex motion artifacts. Different from the previous rPPG models designed for a constant distance between camera and participants, in this paper, we propose two plug-and-play blocks (i.e.,…
West Virginia University（西弗吉尼亚大学）；Indraprastha Institute of Information Technology Delhi（德里英迪拉普拉斯信息技术学院）
Kinship verification has a number of applications such as organizing large collections of images and recognizing resemblances among humans. In this research, first, a human study is conducted to understand the capabilities of human mind and to identify the discriminatory areas of a face that facilitate kinship-cues. Utilizing the information obtained from the human study, a hierarchical Kinship Verification via Representation Learning (KVRL) framework is utilized to learn the representation of different face regions in an unsupervised manner. We propose a novel approach for feature representat…
Dual-path TokenLearner for Remote Photoplethysmography-based Physiological Measurement with Facial Videos
Hefei University of Technology（合肥工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；The 38th Research Institute of China Electronics Technology Group Corporation（中国电子科技集团公司第三十八研究所）
Kinship verification from facial images has been recognized as an emerging yet challenging technique in many potential computer vision applications. In this paper, we propose a novel cross-generation feature interaction learning (CFIL) framework for robust kinship verification. Particularly, an effective collaborative weighting strategy is constructed to explore the characteristics of cross-generation relations by corporately extracting features of both parents and children image pairs. Specifically, we take parents and children as a whole to extract the expressive local and non-local features…
Biometric Authentication Based on Enhanced Remote Photoplethysmography Signal Morphology
Center for Machine Vision and Signal Analysis, University of Oulu（奥卢大学机器视觉与信号分析中心）；State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）
University of Oulu（奥卢大学）；VTT Technical Research Centre of Finland Ltd（芬兰VTT技术研究中心有限公司）
Depression is a mental illness that may be harmful to an individual's health. The detection of mental health disorders in the early stages and a precise diagnosis are critical to avoid social, physiological, or psychological side effects. This work analyzes physiological signals to observe if different depressive states have a noticeable impact on the blood volume pulse (BVP) and the heart rate variability (HRV) response. Although typically, HRV features are calculated from biosignals obtained with contact-based sensors such as wearables, we propose instead a novel scheme that directly extract…
PhysFlow: Frequency Decoupled with Dual-Field Rectified Flow for Remote Photoplethysmography
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院教育部高维信息智能感知与系统重点实验室PCA实验室）；College of Computer Science and Technology, Qingdao University（青岛大学计算机科学技术学院）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Great Bay University（大湾区大学）；Hefei University of Technology（合肥工业大学）；Macao Polytechnic University（澳门理工学院）
Architectural improvements are studied for convolutional network performing estimation of heart rate (HR) values on color signal patches. Color signals are time series of color components averaged over facial regions recorded by webcams in two scenarios: Stationary (without motion of a person) and Mixed Motion (different motion patterns of a person). HR estimation problem is addressed as a classification task, where classes correspond to different heart rate values within the admissible range of [40; 125] bpm. Both adding convolutional filtering layers after fully connected layers and involvin…
3D mask face presentation attack detection (PAD) plays a vital role in securing face recognition systems from emergent 3D mask attacks. Recently, remote photoplethysmography (rPPG) has been developed as an intrinsic liveness clue for 3D mask PAD without relying on the mask appearance. However, the rPPG features for 3D mask PAD are still needed expert knowledge to design manually, which limits its further progress in the deep learning and big data era. In this letter, we propose a pure rPPG transformer (TransRPPG) framework for learning intrinsic liveness representation efficiently. At first, r…
Measurement of the cardiac pulse from facial video has become an interesting pursuit of research over the last few years. This is mainly due to the increasing importance of obtaining the heart rate of an individual in a non-invasive manner, which can be highly useful for applications in gaming and the medical industry. Another instrumental area of research over the past few years has been the advent of Deep Learning and using Deep Neural networks to enhance task performance. In this work, we propose to use efficient convolutional networks to accurately measure the heart rate of user from low r…
Recent studies demonstrated that the average heart rate (HR) can be measured from facial videos based on non-contact remote photoplethysmography (rPPG). However for many medical applications (e.g., atrial fibrillation (AF) detection) knowing only the average HR is not sufficient, and measuring precise rPPG signals from face for heart rate variability (HRV) analysis is needed. Here we propose an rPPG measurement method, which is the first work to use deep spatio-temporal networks for reconstructing precise rPPG signals from raw facial videos. With the constraint of trend-consistency with ground…
HOT: Harmonic-Constrained Optimal Transport for Remote Photoplethysmography Domain Adaptation
VNU University of Engineering and Technology, Hanoi, Vietnam（越南河内国立大学工程与技术大学）；VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam（VinUni-Illinois智慧健康中心，VinUniversity，越南河内）；College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam（VinUniversity工程与计算机科学学院，越南河内）；Center for Innovations in Health Sciences, VinUniversity, Hanoi, Vietnam（VinUniversity健康科学创新中心，越南河内）
FreqPhys: Repurposing Implicit Physiological Frequency Prior for Robust Remote Photoplethysmography
Hefei University of Technology（合肥工业大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Science and Technology Beijing（北京科技大学）；Great Bay University（大湾区大学）
Southeast University（东南大学）；Northeastern University（东北大学）；University of California San Diego（加州大学圣地亚哥分校）；ShangHai Jiao Tong University（上海交通大学）；University of Massachusetts Dartmouth（马萨诸塞大学达特茅斯分校）
Visual kinship recognition aims to identify blood relatives from facial images. Its practical application-- like in law-enforcement, video surveillance, automatic family album management, and more-- has motivated many researchers to put forth effort on the topic as of recent. In this paper, we focus on a new view of visual kinship technology: kin-based face generation. Specifically, we propose a two-stage kin-face generation model to predict the appearance of a child given a pair of parents. The first stage includes a deep generative adversarial autoencoder conditioned on ages and genders to m…
Remote photoplethysmography (rPPG) allows for noncontact monitoring of blood volume changes from a camera by detecting minor fluctuations in reflected light. Prior applications of rPPG focused on face videos. In this paper we explored the feasibility of rPPG from non-face body regions such as the arms, legs, and hands. We collected a new dataset titled Multi-Site Physiological Monitoring (MSPM), which will be released with this paper. The dataset consists of 90 frames per second video of exposed arms, legs, and face, along with 10 synchronized PPG recordings. We performed baseline heart rate e…
BTS-rPPG: Orthogonal Butterfly Temporal Shifting for Remote Photoplethysmography
VNU University of Engineering and Technology, Hanoi, Vietnam（越南河内国立大学工程与技术大学）；VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam（VinUni-Illinois智能健康中心，VinUniversity，越南河内）；College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam（VinUniversity工程与计算机科学学院，越南河内）；Center for Innovations in Health Sciences, VinUniversity, Hanoi, Vietnam（VinUniversity健康科学创新中心，越南河内）
PHASE-Net: Physics-Grounded Harmonic Attention System for Efficient Remote Photoplethysmography Measurement
Great Bay University（大湾大学）；Hefei University of Technology（合肥工业大学）；Macao Polytechnic University（澳门 polytechnic 大学）；University of Science and Technology Beijing（北京科技大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳分校）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞智能与信息科技重点实验室）
Remote photoplethysmography (rPPG) is an attractive camera-based health monitoring method that can measure the heart rhythm from facial videos. Many well-established deep-learning models have been reported to measure heart rate (HR) and heart rate variability (HRV). However, most of these models usually require a 30-second facial video and enormous computational resources to obtain accurate and robust results, which significantly limits their applications in real-world scenarios. Hence, we propose a lightweight pulse extraction network, FastBVP-Net, to quickly measure heart rhythm via facial v…
Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
Department of Biomedical Engineering（生物医学工程系）；Anhui Province Key Laboratory of Measuring Theory and Precision Instrument（安徽省测量理论与精密仪器重点实验室）；School of Computer Science and Information Engineering（计算机科学与信息工程学院）
University of Oulu（奥卢大学）；University of Oxford（牛津大学）；Xi’an Jiaotong University（西安交通大学）；The University of Hong Kong（香港大学）
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose the PhysFormer, an end-to-end video transformer based architecture, to adaptively aggregate both loc…
Beihang University（北京航空航天大学）；Zhejiang University of Technology（浙江工业大学）
With the improvement of sensor technology and significant algorithmic advances, the accuracy of remote heart rate monitoring technology has been significantly improved. Despite of the significant algorithmic advances, the performance of rPPG algorithm can degrade in the long-term, high-intensity continuous work occurred in evenings or insufficient light environments. One of the main challenges is that the lost facial details and low contrast cause the failure of detection and tracking. Also, insufficient lighting in video capturing hurts the quality of physiological signal. In this paper, we c…
Great Bay University（大湾区大学）；University of Oxford（牛津大学）；Xi’an Jiaotong University（西安交通大学）；The University of Hong Kong（香港大学）；University of Oulu（奥卢大学）
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptiv…
