We introduce DOPE, the first method to detect and estimate whole-body 3D human poses, including bodies, hands and faces, in the wild. Achieving this level of details is key for a number of applications that require understanding the interactions of the people with each other or with the environment. The main challenge is the lack of in-the-wild data with labeled whole-body 3D poses. In previous work, training data has been annotated or generated for simpler tasks focusing on bodies, hands or faces separately. In this work, we propose to take advantage of these datasets to train independent experts for each part, namely a body, a hand and a face expert, and distill their knowledge into a single deep network designed for whole-body 2D-3D pose detection. In practice, given a training image with partial or no annotation, each part expert detects its subset of keypoints in 2D and 3D and the resulting estimations are combined to obtain whole-body pseudo ground-truth poses. A distillation loss encourages the whole-body predictions to mimic the experts' outputs. Our results show that this approach significantly outperforms the same whole-body model trained without distillation while staying close to the performance of the experts. Importantly, DOPE is computationally less demanding than the ensemble of experts and can achieve real-time performance. Test code and models are available at https://europe.naverlabs.com/research/computer-vision/dope.
Due to the collection of big data and the development of deep learning, research to predict human emotions in the wild is being actively conducted. We designed a multi-task model using ABAW dataset to predict valence-arousal, expression, and action unit through audio data and face images at in real world. We trained model from the incomplete label by applying the knowledge distillation technique. The teacher model was trained as a supervised learning method, and the student model was trained by using the output of the teacher model as a soft label. As a result we achieved 2.40 in Multi Task Le…
Visual Place Recognition in Forests with Depth-Aware Distillation
CSIRO Robotics, Brisbane, Australia（澳大利亚联邦科学与工业研究组织机器人实验室，布里斯班，澳大利亚）；University of Queensland, Brisbane, Australia（昆士兰大学，布里斯班，澳大利亚）；Queensland University of Technology, Brisbane, Australia（昆士兰科技大学，布里斯班，澳大利亚）
Hong Kong University of Science and Technology（香港科技大学）
When recognizing emotions, subtle nuances in displays of emotion generate ambiguity or uncertainty in emotion perception. Emotion uncertainty has been previously interpreted as inter-rater disagreement among multiple annotators. In this paper, we consider a more common and challenging scenario: modeling emotion uncertainty when only single emotion labels are available. From a Bayesian perspective, we propose to use deep ensembles to capture uncertainty for multiple emotion descriptors, i.e., action units, discrete expression labels and continuous descriptors. We further apply iterative self-di…
Presence of noise in the labels of large scale facial expression datasets has been a key challenge towards Facial Expression Recognition (FER) in the wild. During early learning stage, deep networks fit on clean data. Then, eventually, they start overfitting on noisy labels due to their memorization ability, which limits FER performance. This work proposes an effective training strategy in the presence of noisy labels, called as Consensual Collaborative Training (CCT) framework. CCT co-trains three networks jointly using a convex combination of supervision loss and consistency loss, without ma…
FeatDistill: A Feature Distillation Enhanced Multi-Expert Ensemble Framework for Robust AI-generated Image Detection
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学物联网智能城市国家重点实验室）；PRADA Lab, King Abdullah University of Science and Technology（科学与技术王国大学PRADA实验室）；Department of Information Engineering, University of Florence（佛罗伦萨大学信息工程系）
Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences (UCAS)（中国科学院大学先进交叉科学学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Ant Group（蚂蚁集团）；Sangfor Technologies Inc.（深信服科技股份有限公司）
SZABIST（巴基斯坦真纳科技大学）；Technische Universität Dresden (TU Dresden)（德累斯顿工业大学）
Facial expressions are the most common universal forms of body language. In the past few years, automatic facial expression recognition (FER) has been an active field of research. However, it is still a challenging task due to different uncertainties and complications. Nevertheless, efficiency and performance are yet essential aspects for building robust systems. We proposed two models, EmoXNet which is an ensemble learning technique for learning convoluted facial representations, and EmoXNetLite which is a distillation technique that is useful for transferring the knowledge from our ensemble …
Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beihang University（北京航空航天大学）
Amazon Lab126（亚马逊Lab126）；School of Computing, University of Utah（犹他大学计算学院）
Recovering 3D human pose from 2D joints is a highly unconstrained problem. We propose a novel neural network framework, PoseNet3D, that takes 2D joints as input and outputs 3D skeletons and SMPL body model parameters. By casting our learning approach in a student-teacher framework, we avoid using any 3D data such as paired/unpaired 3D data, motion capture sequences, depth images or multi-view images during training. We first train a teacher network that outputs 3D skeletons, using only 2D poses for training. The teacher network distills its knowledge to a student network that predicts 3D pose …
Current lane detection methods are struggling with the invisibility lane issue caused by heavy shadows, severe road mark degradation, and serious vehicle occlusion. As a result, discriminative lane features can be barely learned by the network despite elaborate designs due to the inherent invisibility of lanes in the wild. In this paper, we target at finding an enhanced feature space where the lane features are distinctive while maintaining a similar distribution of lanes in the wild. To achieve this, we propose a novel Repainting and Imitating Learning (RIL) framework containing a pair of tea…
Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling
School of Artificial Intelligence and Computer Science, Shaanxi Normal University（陕西师范大学人工智能与计算机科学学院）；iFLYTEK Research, iFLYTEK Co., Ltd.（科大讯飞股份有限公司）；Key Laboratory of Intelligent Computing and Service Technology for Folk SongMinistry of Culture and Tourism（文化和旅游部民歌智能计算与服务技术重点实验室）
The University of Tokyo（东京大学）；Carnegie Mellon University（卡内基梅隆大学）
We aim to improve the performance of regressing hand keypoints and segmenting pixel-level hand masks under new imaging conditions (e.g., outdoors) when we only have labeled images taken under very different conditions (e.g., indoors). In the real world, it is important that the model trained for both tasks works under various imaging conditions. However, their variation covered by existing labeled hand datasets is limited. Thus, it is necessary to adapt the model trained on the labeled images (source) to unlabeled images (target) with unseen imaging conditions. While self-training domain adapt…
Towards Activated Muscle Group Estimation in the Wild
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Stuttgart University（斯图加特大学）；Hunan University（湖南大学）；Beijing Sport University（北京体育大学）；Mercedes-Benz Tech Innovation（梅赛德斯-奔驰技术创新公司）
Efficient Low-Resolution Face Recognition via Bridge Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；SenseTime Group Limited（商汤集团）；Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）
Dual-stream Transformer-GCN Model with Contextualized Representations Learning for Monocular 3D Human Pose Estimation
Northeastern University（东北大学）；College of Sciences, Northeastern University（东北大学理学院）；Key Laboratory of Differential Equations and Their Applications, Northeastern University（东北大学微分方程及应用重点实验室）；Liaoning Provincial Department of Education（辽宁省教育厅）；King’s College London（伦敦国王学院）；Shenyang sport University（沈阳体育学院）
