Multi-task learning is an effective learning strategy for deep-learning-based facial expression recognition tasks. However, most existing methods take into limited consideration the feature selection, when transferring information between different tasks, which may lead to task interference when training the multi-task networks. To address this problem, we propose a novel selective feature-sharing method, and establish a multi-task network for facial expression recognition and facial expression synthesis. The proposed method can effectively transfer beneficial features between different tasks, while filtering out useless and harmful information. Moreover, we employ the facial expression synthesis task to enlarge and balance the training dataset to further enhance the generalization ability of the proposed method. Experimental results show that the proposed method achieves state-of-the-art performance on those commonly used facial expression recognition benchmarks, which makes it a potential solution to real-world facial expression recognition problems.
Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to t…
FER-YOLO-Mamba: Facial Expression Detection and Classification Based on Selective State Space
School of Information Science and Technology, Southwest Jiaotong University（西南交通大学信息科学与技术学院）；School of Electrical and Information Engineering, University of the Witwatersrand（金山大学电气与信息工程学院）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beihang University（北京航空航天大学）
Typically, the deployment of face recognition models in the wild needs to identify low-resolution faces with extremely low computational cost. To address this problem, a feasible solution is compressing a complex face model to achieve higher speed and lower memory at the cost of minimal performance drop. Inspired by that, this paper proposes a learning approach to recognize low-resolution faces via selective knowledge distillation. In this approach, a two-stream convolutional neural network (CNN) is first initialized to recognize high-resolution faces and resolution-degraded faces with a teach…
Korea Electronics Technology Institute（韩国电子技术研究院）；Chonbuk National University（全北国立大学）；Simon Fraser University（西蒙弗雷泽大学）
Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points …
Semi-Supervised Facial Expression Recognition based on Dynamic Threshold and Negative Learning
University of Science and Technology of China（中国科学技术大学）；Jianghuai Advance Technology Center（江淮前沿技术协同创新中心）；Unisound AI Technology Co., Ltd（云知声智能科技股份有限公司）
Huazhong University of Science and Technology（华中科技大学）
Facial expression recognition (FER), aiming to classify the expression present in the facial image or video, has attracted a lot of research interests in the field of artificial intelligence and multimedia. In terms of video based FER task, it is sensible to capture the dynamic expression variation among the frames to recognize facial expression. However, existing methods directly utilize CNN-RNN or 3D CNN to extract the spatial-temporal features from different facial units, instead of concentrating on a certain region during expression variation capturing, which leads to limited performance i…
On Applicability of Synthetic Datasets for Facial Expression Recognition
Department of Computer Engineering, Istanbul Technical University（伊斯坦布尔技术大学计算机工程系）；Department of Informatics, University of Zurich（苏黎世大学信息学系）；Division of Engineering, NYU Abu Dhabi（纽约大学阿布扎克分校工程系）
Microsoft Research（微软研究院）；Wharton School University of Pennsylvania（宾夕法尼亚大学沃顿商学院）
People often share news and information with their social connections, but why do some advertisements get shared more than others? A large-scale test examines whether facial responses predict sharing. Facial expressions play a key role in emotional expression. Using scalable automated facial coding algorithms, we quantify the facial expressions of thousands of individuals in response to hundreds of advertisements. Results suggest that not all emotions expressed during viewing increase sharing, and that the relationship between emotion and transmission is more complex than mere valence alone. F…
Clemson University（克莱姆森大学）；University of Colorado Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）
Facial expressions convey massive information and play a crucial role in emotional expression. Deep neural network (DNN) accompanied by deep metric learning (DML) techniques boost the discriminative ability of the model in facial expression recognition (FER) applications. DNN, equipped with only classification loss functions such as Cross-Entropy cannot compact intra-class feature variation or separate inter-class feature distance as well as when it gets fortified by a DML supporting loss item. The triplet center loss (TCL) function is applied on all dimensions of the sample's embedding in the…
Content based image retrieval, a technique which uses visual contents of image to search images from large scale image databases according to users' interests. This paper provides a comprehensive survey on recent technology used in the area of content based face image retrieval. Nowadays digital devices and photo sharing sites are getting more popularity, large human face photos are available in database. Multiple types of facial features are used to represent discriminality on large scale human facial image database. Searching and mining of facial images are challenging problems and important…
Camera sensors are increasingly being combined with machine learning to perform various tasks such as intelligent surveillance. Due to its computational complexity, most of these machine learning algorithms are offloaded to the cloud for processing. However, users are increasingly concerned about privacy issues such as function creep and malicious usage by third-party cloud providers. To alleviate this, we propose an edge-based filtering stage that removes privacy-sensitive attributes before the sensor data are transmitted to the cloud. We use state-of-the-art image manipulation techniques tha…
High-performance visual recognition systems generally require a large collection of labeled images to train. The expensive data curation can be an obstacle for improving recognition performance. Sharing more data allows training for better models. But personal and private information in the data prevent such sharing. To promote sharing visual data for learning a recognition model, we propose to obfuscate the images so that humans are not able to recognize their detailed contents, while machines can still utilize them to train new models. We validate our approach by comprehensive experiments on…
CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61）；Cyber Security Cooperative Research Centre（网络安全合作研究中心）
Biometric data, such as face images, are often associated with sensitive information (e.g medical, financial, personal government records). Hence, a data breach in a system storing such information can have devastating consequences. Deep learning is widely utilized for face recognition (FR); however, such models are vulnerable to backdoor attacks executed by malicious parties. Backdoor attacks cause a model to misclassify a particular class as a target class during recognition. This vulnerability can allow adversaries to gain access to highly sensitive data protected by biometric authenticatio…
Convolutional neural networks (CNNs) have become the most successful approach in many vision-related domains. However, they are limited to domains where data is abundant. Recent works have looked at multi-task learning (MTL) to mitigate data scarcity by leveraging domain-specific information from related tasks. In this paper, we present a novel soft-parameter sharing mechanism for CNNs in a MTL setting, which we refer to as Deep Collaboration. We propose taking into account the notion that task relevance depends on depth by using lateral transformation blocs with skip connections. This allows …
Massachusetts Institute of Technology（麻省理工学院）；Harvard Medical School（哈佛医学院）
Recent deep learning models have shown remarkable performance in image classification. While these deep learning systems are getting closer to practical deployment, the common assumption made about data is that it does not carry any sensitive information. This assumption may not hold for many practical cases, especially in the domain where an individual's personal information is involved, like healthcare and facial recognition systems. We posit that selectively removing features in this latent space can protect the sensitive information and provide a better privacy-utility trade-off. Consequen…
Anonymization of medical images is necessary for protecting the identity of the test subjects, and is therefore an essential step in data sharing. However, recent developments in deep learning may raise the bar on the amount of distortion that needs to be applied to guarantee anonymity. To test such possibilities, we have applied the novel CycleGAN unsupervised image-to-image translation framework on sagittal slices of T1 MR images, in order to reconstruct facial features from anonymized data. We applied the CycleGAN framework on both face-blurred and face-removed images. Our results show that…
Samsung Semiconductor Israel R&D Center (SIRC)（三星半导体以色列研发中心（SIRC））；Faculty of Engineering, Bar-Ilan University（巴伊兰大学工程学院）
Facial landmarks (FLM) estimation is a critical component in many face-related applications. In this work, we aim to optimize for both accuracy and speed and explore the trade-off between them. Our key observation is that not all faces are created equal. Frontal faces with neutral expressions converge faster than faces with extreme poses or expressions. To differentiate among samples, we train our model to predict the regression error after each iteration. If the current iteration is accurate enough, we stop iterating, saving redundant iterations while keeping the accuracy in check. We also ob…
In contrast to great success of memory-consuming face editing methods at a low resolution, to manipulate high-resolution (HR) facial images, i.e., typically larger than 7682 pixels, with very limited memory is still challenging. This is due to the reasons of 1) intractable huge demand of memory; 2) inefficient multi-scale features fusion. To address these issues, we propose a NOVEL pixel translation framework called Cooperative GAN(CooGAN) for HR facial image editing. This framework features a local path for fine-grained local facial patch generation (i.e., patch-level HR, LOW memory) and a gl…
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（机器人与工业信息研究所（CSIC-UPC））；The Ohio State University（俄亥俄州立大学）
Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for task of facial expression synthesis. The most successful architecture is StarGAN, that conditions GANs generation process with images of a specific domain, namely a set of images of persons sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous mani…
University of Science and Technology of China（中国科学技术大学）
Facial valence/arousal, expression and action unit are related tasks in facial affective analysis. However, the tasks only have limited performance in the wild due to the various collected conditions. The 4th competition on affective behavior analysis in the wild (ABAW) provided images with valence/arousal, expression and action unit labels. In this paper, we introduce multi-task learning framework to enhance the performance of three related tasks in the wild. Feature sharing and label fusion are used to utilize their relations. We conduct experiments on the provided training and validating da…
FED-PsyAU: Privacy-Preserving Micro-Expression Recognition via Psychological AU Coordination and Dynamic Facial Motion Modeling
Institute of Psychology, Chinese Academy of Sciences（中国科学院心理研究所）；University of the Chinese Academy of Sciences（中国科学院大学）；Jiangsu University of Science and Technology（江苏科技大学）
针对微表情样本少、特征细微及数据隐私问题，提出 FED-PsyAU 框架，结合心理 AU 协同先验、DPK-GAT 分层运动建模与联邦学习，在保护隐私的同时提升多客户端识别性能。
IIIT Delhi（德里印度理工学院）；Ministry of Electronics and Information Technology（电子和信息技术部）
A face image not only provides details about the identity of a subject but also reveals several attributes such as gender, race, sexual orientation, and age. Advancements in machine learning algorithms and popularity of sharing images on the World Wide Web, including social media websites, have increased the scope of data analytics and information profiling from photo collections. This poses a serious privacy threat for individuals who do not want to be profiled. This research presents a novel algorithm for anonymizing selective attributes which an individual does not want to share without aff…
Automatic neural architecture search techniques are becoming increasingly important in machine learning area. Especially, weight sharing methods have shown remarkable potentials on searching good network architectures with few computational resources. However, existing weight sharing methods mainly suffer limitations on searching strategies: these methods either uniformly train all network paths to convergence which introduces conflicts between branches and wastes a large amount of computation on unpromising candidates, or selectively train branches with different frequency which leads to unfa…
University of Cambridge（剑桥大学）；Middle East Technical University（中东技术大学）
Most state-of-the-art approaches for Facial Action Unit (AU) detection rely upon evaluating facial expressions from static frames, encoding a snapshot of heightened facial activity. In real-world interactions, however, facial expressions are usually more subtle and evolve in a temporal manner requiring AU detection models to learn spatial as well as temporal information. In this paper, we focus on both spatial and spatio-temporal features encoding the temporal evolution of facial AU activation. For this purpose, we propose the Action Unit Lifecycle-Aware Capsule Network (AULA-Caps) that perfor…
In this paper, we propose a computational efficient end-to-end training deep neural network (CEDNN) model and spatial attention maps based on difference images. Firstly, the difference image is generated by image processing. Then five binary images of difference images are obtained using different thresholds, which are used as spatial attention maps. We use group convolution to reduce model complexity. Skip connection and $\text{1}\times \text{1}$ convolution are used to ensure good performance even if the network model is not deep. As an input, spatial attention map can be selectively fed int…
Real-Time Multi-Modal Embedded Vision Framework for Object Detection Facial Emotion Recognition and Biometric Identification on Low-Power Edge Platforms
1,2Department of Mechatronics Engineering, Rajshahi University of Engineering \& Technology 3 ,4,5Department of Internet of Things；Robotics Engineering, University of Frontier Technology, Bangladesh 6Department of Computer Science；Engineering, Varendra University, Rajshahi, Bangladesh 7Department of Robotics, Robo Tech Valley, Dhaka, Bangladesh Emails: , , 3
Tsinghua University（清华大学）；National Engineering Laboratory for Neuromodulation（神经调控国家工程实验室）
Data sharing for medical research has been difficult as open-sourcing clinical data may violate patient privacy. Traditional methods for face de-identification wipe out facial information entirely, making it impossible to analyze facial behavior. Recent advancements on whole-body keypoints detection also rely on facial input to estimate body keypoints. Both facial and body keypoints are critical in some medical diagnoses, and keypoints invariability after de-identification is of great importance. Here, we propose a solution using deepfake technology, the face swapping technique. While this swa…
Hunan University（湖南大学）；National University of Defense Technology（国防科技大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Face multi-attribute prediction benefits substantially from multi-task learning (MTL), which learns multiple face attributes simultaneously to achieve shared or mutually related representations of different attributes. The most widely used MTL convolutional neural network is heuristically or empirically designed by sharing all of the convolutional layers and splitting at the fully connected layers for task-specific losses. However, it is improper to view all low and mid-level features for different attributes as being the same, especially when these attributes are only loosely related. In this…
ResEmoteNet: Bridging Accuracy and Loss Reduction in Facial Emotion Recognition
Sikkim Manipal Institute of Technology (SMIT)（锡金曼尼普尔理工学院（SMIT））；National Institute of Technology Sikkim（锡金国家理工学院）；Bay Area Advanced Analytics India (P) Ltd（湾区高级分析印度私人有限公司）
Accurate and Private Diagnosis of Rare Genetic Syndromes from Facial Images with Federated Deep Learning
Medical Data Privacy and Privacy Preserving Machine Learning (MDPPML)（医学数据隐私与隐私保护机器学习）；University of Tübingen（图宾根大学）；Institute for Bioinformatics and Medical Informatics (IBMI)（生物信息学与医学信息学研究所）；Institute for Genomic Statistics and Bioinformatics（基因组统计与生物信息学研究所）；University of Bonn（波恩大学）
UC San Diego（加州大学圣地亚哥分校）；IBM Research（IBM研究院）；Binghamton Univeristy, SUNY（纽约州立大学宾汉姆顿大学）
Multi-task learning aims to improve generalization performance of multiple prediction tasks by appropriately sharing relevant information across them. In the context of deep neural networks, this idea is often realized by hand-designed network architectures with layers that are shared across tasks and branches that encode task-specific features. However, the space of possible multi-task deep architectures is combinatorially large and often the final architecture is arrived at by manual exploration of this space subject to designer's bias, which can be both error-prone and tedious. In this work…
MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
School of Computer Science and Technology, China University of Mining and Technology（中国矿业大学（北京）计算机科学与技术学院）；Mine Digitization Engineering Research Center of the Ministry of Education（教育部矿山数字化工程研究中心）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Department of Computer Science and Software Engineering, The University of Western Australia（西澳大学计算机科学与软件工程系）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能教育部重点实验室）
PhysVR: Vision-Language Model Guided Interference-aware Temporal Feature Refinement for Remote Physiological Measurement
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；College of Computer Science and Technology, Qingdao University（青岛大学计算机科学技术学院）
Privacy-Preserving Federated Action Recognition via Differentially Private Selective Tuning and Efficient Communication
School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；Infocomm Technology Cluster, Singapore Institute of Technology（新加坡理工学院信息通信技术集群）；The Edward S. Rogers Sr. Dept. of Electrical and Computer Engineering, University of Toronto（多伦多大学埃德华·S·罗杰斯Sr.电子与计算机工程系）
Southeast University（东南大学）；South China University of Technology（华南理工大学）；University of Oulu（奥卢大学）；Nanjing University of Science and Technology（南京理工大学）
Cross-database micro-expression recognition (CDMER) is one of recently emerging and interesting problem in micro-expression analysis. CDMER is more challenging than the conventional micro-expression recognition (MER), because the training and testing samples in CDMER come from different micro-expression databases, resulting in the inconsistency of the feature distributions between the training and testing sets. In this paper, we contribute to this topic from three aspects. First, we establish a CDMER experimental evaluation protocol aiming to allow the researchers to conveniently work on this …
University of Jinan（济南大学）；Jiangnan University（江南大学）
Existing multi-scale solutions lead to a risk of just increasing the receptive field sizes while neglecting small receptive fields. Thus, it is a challenging problem to effectively construct adaptive neural networks for recognizing various spatial-scale objects. To tackle this issue, we first introduce a new attention dimension, i.e., depth, in addition to existing attention dimensions such as channel, spatial, and branch, and present a novel selective depth attention network to symmetrically handle multi-scale objects in various vision tasks. Specifically, the blocks within each stage of a gi…
