23 pages, 17 figures, 20 tables. Accepted by IEEE Transactions on Affective Computing. Revised after peer review. The source code and models will be publicly available at https://github.com/MSA-LMC/SSM
面部动作单元（AU）检测和面部表情（FE）识别可视为情感面部行为任务，分别代表细粒度的肌肉激活和粗粒度的整体情感状态。尽管它们存在内在语义相关性，现有研究主要关注从AU到FE的知识迁移，而双向学习仍不足。实际中，异构数据条件进一步加剧了这一挑战，因为AU和FE数据集在标注范式（帧级 vs. 视频级）、标签粒度和数据可用性与多样性上存在差异，阻碍了有效的联合学习。为了解决这些问题，我们提出了一种结构化语义映射（SSM）框架，用于在不同数据域和异构监督下实现双向AU-FE学习。SSM包含三个关键组件：（1）一个共享的视觉主干，从动态AU和FE视频中学习统一的面部表示；（2）通过文本语义原型（TSP）模块进行语义中介，该模块从固定文本描述中构建结构化的语义原型，并辅以可学习的上下文提示，作为监督信号和跨任务对齐的锚点；（3）动态先验映射（DPM）模块，该模块从面部动作编码系统中提取先验知识，并在高层特征空间中学习数据驱动的关联矩阵，实现显式和双向的知识迁移。在流行的AU检测和FE识别基准上的广泛实验表明，SSM在两项任务上均取得最佳性能，并证明整体表情语义可以反过来增强细粒度AU学习，即使在异构数据集上。
英文摘要：
Facial action unit (AU) detection and facial expression (FE) recognition can be jointly viewed as affective facial behavior tasks, representing fine-grained muscular activations and coarse-grained holistic affective states, respectively. Despite their inherent semantic correlation, existing studies predominantly focus on knowledge transfer from AUs to FEs, while bidirectional learning remains insufficiently explored. In practice, this challenge is further compounded by heterogeneous data conditions, where AU and FE datasets differ in annotation paradigms (frame-level vs.\ clip-level), label granularity, and data availability and diversity, hindering effective joint learning. To address these issues, we propose a Structured Semantic Mapping (SSM) framework for bidirectional AU--FE learning under different data domains and heterogeneous supervision. SSM consists of three key components: (1) a shared visual backbone that learns unified facial representations from dynamic AU and FE videos; (2) semantic mediation via a Textual Semantic Prototype (TSP) module, which constructs structured semantic prototypes from fixed textual descriptions with learnable context prompts for supervision and cross-task alignment in a shared semantic space; and (3) a Dynamic Prior Mapping (DPM) module that incorporates FACS-derived prior knowledge and learns data-adaptive bidirectional association matrices in the textual semantic space for explicit knowledge transfer. Extensive experiments on popular AU detection and FE recognition benchmarks show that SSM consistently outperforms its single-task and multi-task baselines and achieves competitive performance against task-specific methods. The FE-to-AU results further show that holistic expression semantics provides useful supervision for fine-grained AU learning across heterogeneous datasets.
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院整形外科医院）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（香港科学创新研究院人工智能与机器人中心，中国科学院）
Age estimation from facial images is typically cast as a nonlinear regression problem. The main challenge of this problem is the facial feature space w.r.t. ages is heterogeneous, due to the large variation in facial appearance across different persons of the same age and the non-stationary property of aging patterns. In this paper, we propose Deep Regression Forests (DRFs), an end-to-end model, for age estimation. DRFs connect the split nodes to a fully connected layer of a convolutional neural network (CNN) and deal with heterogeneous data by jointly learning input-dependant data partitions …
Jawaharlal Nehru University（贾瓦哈拉尔·尼赫鲁大学）；Indian Institute of Technology Delhi（印度理工学院德里分校）
Predicting attributes in the landmark free facial images is itself a challenging task which gets further complicated when the face gets occluded due to the usage of masks. Smart access control gates which utilize identity verification or the secure login to personal electronic gadgets may utilize face as a biometric trait. Particularly, the Covid-19 pandemic increasingly validates the essentiality of hygienic and contactless identity verification. In such cases, the usage of masks become more inevitable and performing attribute prediction helps in segregating the target vulnerable groups from …
FreqFLD: Towards All-in-One Facial Landmark Detection via Frequency Modulation
China Three Gorges University（三峡大学）；Zhongnan University of Economics and Law（中南财经政法大学）；National Institute of Natural Hazards, Ministry of Emergency Management of China（中国应急管理部国家自然灾害防治研究院）；Yunnan University（云南大学）；Wuhan University（武汉大学）；Nanyang Technological University（南洋理工大学）
Faces of the Mind: Unveiling Mental Health States Through Facial Expressions in 11,427 Adolescents
The Affiliated Brain Hospital of Nanjing Medical University, and the School of Biomedical Engineering and Informatics of Nanjing Medical University（南京医科大学附属脑科医院及南京医科大学生物医学工程与信息学院）
Active Appearance Model (AAM) is a commonly used method for facial image analysis with applications in face identification and facial expression recognition. This paper proposes a new approach based on image alignment for AAM fitting called bidirectional warping. Previous approaches warp either the input image or the appearance template. We propose to warp both the input image, using incremental update by an affine transformation, and the appearance template, using an inverse compositional approach. Our experimental results on Multi-PIE face database show that the bidirectional approach outper…
MSD-KMamba: Bidirectional Spatial-Aware Multi-Modal 3D Brain Segmentation via Multi-scale Self-Distilled Fusion Strategy
Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University（智能计算与信号处理重点实验室，教育部，安徽大学）；Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology（能源化工过程智能制造重点实验室，教育部，东华大学）
Beyond Words: AuralLLM and SignMST-C for Sign Language Production and Bidirectional Accessibility
School of Artificial Intelligence and Advanced Computing, Xi’an Jiaotong-Liverpool University（西交利物浦大学人工智能与高级计算学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Monash University（莫纳什大学）
Lightweight Facial Landmark Detection in Thermal Images via Multi-Level Cross-Modal Knowledge Transfer
Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, Genoa, Italy（人机交互实验室，意大利技术研究院，热那亚，意大利）；Ph.D. Program of National Interest in Robotics and Intelligent Machines (DRIM), Università di Genova（机器人与智能机器国家利益博士项目，热那亚大学）
Semantic face editing has achieved substantial progress in recent years. Known as a growingly popular method, latent space manipulation performs face editing by changing the latent code of an input face to liberate users from painting skills. However, previous latent space manipulation methods usually encode an entire face into a single low-dimensional embedding, which constrains the reconstruction capacity and the control flexibility of facial components, such as eyes and nose. This paper proposes IA-FaceS as a bidirectional method for disentangled face attribute manipulation as well as flexi…
We propose an end to end deep learning approach for generating real-time facial animation from just audio. Specifically, our deep architecture employs deep bidirectional long short-term memory network and attention mechanism to discover the latent representations of time-varying contextual information within the speech and recognize the significance of different information contributed to certain face status. Therefore, our model is able to drive different levels of facial movements at inference and automatically keep up with the corresponding pitch and latent speaking style in the input audio…
Bidirectional Regression for Monocular 6DoF Head Pose Estimation and Reference System Alignment
Department of Electronics and Communications Engineering, Kwangwoon University（韩国 Kwangwoon 大学电子与通信工程系）；Department of Software Engineering, Dankook University（韩国 Dankook 大学软件工程系）；School of Computer Science, University of Birmingham（英国 Birmingham 大学计算机科学学院）
Fujitsu Research and Development Center（富士通研究开发中心）；Fujitsu Laboratories Ltd.（富士通株式会社）
The detection of facial action units (AUs) has been studied as it has the competition due to the wide-ranging applications thereof. In this paper, we propose a novel framework for the AU detection from a single input image by grasping the \textbf{c}o-\textbf{o}ccurrence and \textbf{m}utual \textbf{ex}clusion (COMEX) as well as the intensity distribution among AUs. Our algorithm uses facial landmarks to detect the features of local AUs. The features are input to a bidirectional long short-term memory (BiLSTM) layer for learning the intensity distribution. Afterwards, the new AU feature continuo…
Yonsei University（延世大学）；Korea Institute of Science and Technology（韩国科学技术研究院）；Yongin University（龙仁大学）
Heterogeneous Face Recognition (HFR) is a task that matches faces across two different domains such as visible light (VIS), near-infrared (NIR), or the sketch domain. Due to the lack of databases, HFR methods usually exploit the pre-trained features on a large-scale visual database that contain general facial information. However, these pre-trained features cause performance degradation due to the texture discrepancy with the visual domain. With this motivation, we propose a graph-structured module called Relational Graph Module (RGM) that extracts global relational information in addition to …
FacialPulse: An Efficient RNN-based Depression Detection via Temporal Facial Landmarks
Hefei University of Technology（合肥工业大学）；CFAR（计算科学与工程中心）；IHPC（高性能计算研究所）；A*STAR（新加坡科技研究局）；University of Science and Technology of China（中国科学技术大学）；The University of Electro-Communications（东京电机大学）
A$^{3}$lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP
Shanghai Engineering Research Center of AI & Robotics（上海人工智能与机器人工程研究中心）；Academy for Engineering & Technology, Fudan University（复旦大学工程与技术学院）；School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）
National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Center for Excellence in Brain Science and Intelligence Technology, CAS（中国科学院脑科学与智能技术卓越创新中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
We propose a new challenging task namely IDentity Stylization (IDS) across heterogeneous domains. IDS focuses on stylizing the content identity, rather than completely swapping it using the reference identity. We use an effective heterogeneous-network-based framework $Styleverse$ that uses a single domain-aware generator to exploit the Metaverse of diverse heterogeneous faces, based on the proposed dataset FS13 with limited data. FS13 means 13 kinds of Face Styles considering diverse lighting conditions, art representations and life dimensions. Previous similar tasks, \eg, image style transfer…
Lanzhou University（兰州大学）；University of Guelph（圭尔夫大学）；Zhejiang Sci-Tech University（浙江理工大学）；Chongqing University（重庆大学）
The ever-increasing demands for intuitive interactions in Virtual Reality has triggered a boom in the realm of Facial Expression Recognition (FER). To address the limitations in existing approaches (e.g., narrow receptive fields and homogenous supervisory signals) and further cement the capacity of FER tools, a novel multifarious supervision-steering Transformer for FER in the wild is proposed in this paper. Referred as FER-former, our approach features multi-granularity embedding integration, hybrid self-attention scheme, and heterogeneous domain-steering supervision. In specific, to dig deep…
AU-TTT: Vision Test-Time Training model for Facial Action Unit Detection
Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Great Bay University（大湾区大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；Brno University of Technology（布尔诺理工大学）
SenseTime Group Limited（商汤集团有限公司）；The Chinese University of Hong Kong（香港中文大学）
Interpersonal relation defines the association, e.g., warm, friendliness, and dominance, between two or more people. Motivated by psychological studies, we investigate if such fine-grained and high-level relation traits can be characterized and quantified from face images in the wild. We address this challenging problem by first studying a deep network architecture for robust recognition of facial expressions. Unlike existing models that typically learn from facial expression labels alone, we devise an effective multitask network that is capable of learning from rich auxiliary attributes such …
Sorbonne Universités（索邦大学）；UPMC Univ Paris 06（巴黎第六大学）；CNRS（法国国家科学研究中心）；ISIR UMR 7222（智能系统与机器人研究所）
Automatic facial expression classification (FER) from videos is a critical problem for the development of intelligent human-computer interaction systems. Still, it is a challenging problem that involves capturing high-dimensional spatio-temporal patterns describing the variation of one's appearance over time. Such representation undergoes great variability of the facial morphology and environmental factors as well as head pose variations. In this paper, we use Conditional Random Forests to capture low-level expression transition patterns. More specifically, heterogeneous derivative features (e…
FauForensics: Boosting Audio-Visual Deepfake Detection with Facial Action Units
School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机科学学院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Tencent AI Lab（腾讯人工智能实验室）
One-shot video-driven talking face generation aims at producing a synthetic talking video by transferring the facial motion from a video to an arbitrary portrait image. Head pose and facial expression are always entangled in facial motion and transferred simultaneously. However, the entanglement sets up a barrier for these methods to be used in video portrait editing directly, where it may require to modify the expression only while maintaining the pose unchanged. One challenge of decoupling pose and expression is the lack of paired data, such as the same pose but different expressions. Only a…
Automated emotion recognition in the wild from facial images remains a challenging problem. Although recent advances in Deep Learning have supposed a significant breakthrough in this topic, strong changes in pose, orientation and point of view severely harm current approaches. In addition, the acquisition of labeled datasets is costly, and current state-of-the-art deep learning algorithms cannot model all the aforementioned difficulties. In this paper, we propose to apply a multi-task learning loss function to share a common feature representation with other related tasks. Particularly we show…
University of Greenwich（格林威治大学）；University of Sussex（萨塞克斯大学）；Imperial College London（伦敦帝国理工学院）
Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm, such as a DNN. MTL is based on the assumption that the tasks under consideration are related; therefore it exploits shared knowledge for improving performance on each individual task. Tasks are generally considered to be homogeneous, i.e., to refer to the same type of problem. Moreover, MTL is usually based on ground truth annotations with full, or partial overlap across tasks. In this work, we deal with heterogeneous MTL, simultaneously addressing detection, classificat…
Istituto di Scienza e Tecnologie dell’Informazione “Alessandro Faedo” – CNR（意大利国家研究委员会信息科学与技术研究所“亚历山德罗·法埃多”）
Emotions play a central role in the social life of every human being, and their study, which represents a multidisciplinary subject, embraces a great variety of research fields. Especially concerning the latter, the analysis of facial expressions represents a very active research area due to its relevance to human-computer interaction applications. In such a context, Facial Expression Recognition (FER) is the task of recognizing expressions on human faces. Typically, face images are acquired by cameras that have, by nature, different characteristics, such as the output resolution. It has been …
Over the centuries, humans have developed and acquired a number of ways to communicate. But hardly any of them can be as natural and instinctive as facial expressions. On the other hand, neural networks have taken the world by storm. And no surprises, that the area of Computer Vision and the problem of facial expressions recognitions hasn't remained untouched. Although a wide range of techniques have been applied, achieving extremely high accuracies and preparing highly robust FER systems still remains a challenge due to heterogeneous details in human faces. In this paper, we will be deep divi…
École de technologie supérieure（高等技术学院）；Concordia University（康考迪亚大学）
Training deep learning models for accurate spatiotemporal recognition of facial expressions in videos requires significant computational resources. For practical reasons, 3D Convolutional Neural Networks (3D CNNs) are usually trained with relatively short clips randomly extracted from videos. However, such uniform sampling is generally sub-optimal because equal importance is assigned to each temporal clip. In this paper, we present a strategy for efficient video-based training of 3D CNNs. It relies on softmax temporal pooling and a weighted sampling mechanism to select the most relevant traini…
In Heterogeneous Face Recognition (HFR), the objective is to match faces across two different domains such as visible and thermal. Large domain discrepancy makes HFR a difficult problem. Recent methods attempting to fill the gap via synthesis have achieved promising results, but their performance is still limited by the scarcity of paired training data. In practice, large-scale heterogeneous face data are often inaccessible due to the high cost of acquisition and annotation process as well as privacy regulations. In this paper, we propose a new face hallucination paradigm for HFR, which not on…
Bi-CRCL: Bidirectional Conservative-Radical Complementary Learning with Pre-trained Foundation Models for Class-incremental Medical Image Analysis
Department of Biomedical Engineering, The Chinese University of Hong Kong（生物医学工程系，香港中文大学）；Department of Radiation Oncology, Columbia University Irving Medical Center and Data Science Institute, Columbia University（放射肿瘤学系，哥伦比亚大学伊万杰琳医学中心及数据科学研究院，哥伦比亚大学）；Department of Electrical and Electronic Engineering and School of Biomedical Engineering, The University of Hong Kong（电气电子工程系和生物医学工程学院，香港大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，城市大学）；Department of Artificial Intelligence, Westlake University（人工智能系，西湖大学）
Shanghai Jiao Tong University（上海交通大学）；University of Electronic Science and Technology of China（电子科技大学）；ByteDance AI Lab（字节跳动人工智能实验室）；Nanyang Technological University（南洋理工大学）；University of Guelph（圭尔夫大学）；Memorial University of Newfoundland（纽芬兰纪念大学）
The task of few-shot visual dubbing focuses on synchronizing the lip movements with arbitrary speech input for any talking head video. Albeit moderate improvements in current approaches, they commonly require high-quality homologous data sources of videos and audios, thus causing the failure to leverage heterogeneous data sufficiently. In practice, it may be intractable to collect the perfect homologous data in some cases, for example, audio-corrupted or picture-blurry videos. To explore this kind of data and support high-fidelity few-shot visual dubbing, in this paper, we novelly propose a si…
Accurate and Private Diagnosis of Rare Genetic Syndromes from Facial Images with Federated Deep Learning
Medical Data Privacy and Privacy Preserving Machine Learning (MDPPML)（医学数据隐私与隐私保护机器学习）；University of Tübingen（图宾根大学）；Institute for Bioinformatics and Medical Informatics (IBMI)（生物信息学与医学信息学研究所）；Institute for Genomic Statistics and Bioinformatics（基因组统计与生物信息学研究所）；University of Bonn（波恩大学）
Deep learning models need large amounts of data for training. In video recognition and classification, significant advances were achieved with the introduction of new large databases. However, the creation of large-databases for training is infeasible in several scenarios. Thus, existing or small collected databases are typically joined and amplified to train these models. Nevertheless, training neural networks on limited data is not straightforward and comes with a set of problems. In this paper, we explore the effects of stacking databases, model initialization, and data amplification techni…
University of Florence（佛罗伦萨大学）；University of Siena（锡耶纳大学）
Recognizing facial expressions from static images or video sequences is a widely studied but still challenging problem. The recent progresses obtained by deep neural architectures, or by ensembles of heterogeneous models, have shown that integrating multiple input representations leads to state-of-the-art results. In particular, the appearance and the shape of the input face, or the representations of some face parts, are commonly used to boost the quality of the recognizer. This paper investigates the application of Convolutional Neural Networks (CNNs) with the aim of building a versatile rec…
IIT Bombay（印度理工学院孟买分校）；Cornell University（康奈尔大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Adobe Research（Adobe研究院）
We present a deep, bidirectional, recurrent framework for cleaning noisy and incomplete motion capture data. It exploits temporal coherence and joint correlations to infer adaptive filters for each joint in each frame. A single model can be trained to denoise a heterogeneous mix of action types, under substantial amounts of noise. A signal that has both noise and gaps is preprocessed with a second bidirectional network that synthesizes missing frames from surrounding context. The approach handles a wide variety of noise types and long gaps, does not rely on knowledge of the noise distribution,…
The 3D Morphable Model (3DMM) is a powerful statistical tool for representing 3D face shapes. To build a 3DMM, a training set of face scans in full point-to-point correspondence is required, and its modeling capabilities directly depend on the variability contained in the training data. Thus, to increase the descriptive power of the 3DMM, establishing a dense correspondence across heterogeneous scans with sufficient diversity in terms of identities, ethnicities, or expressions becomes essential. In this manuscript, we present a fully automatic approach that leverages a 3DMM to transfer its den…
Visible-to-thermal face image matching is a challenging variate of cross-modality recognition. The challenge lies in the large modality gap and low correlation between visible and thermal modalities. Existing approaches employ image preprocessing, feature extraction, or common subspace projection, which are independent problems in themselves. In this paper, we propose an end-to-end framework for cross-modal face recognition. The proposed algorithm aims to learn identity-discriminative features from unprocessed facial images and identify cross-modal image pairs. A novel Unit-Class Loss is propo…
