Estimating3D hand poses from RGB images is essentialto a wide range of potential applications, but is challengingowing to substantial ambiguity in the inference of depth in-formation from RGB images. State-of-the-art estimators ad-dress this problem by regularizing3D hand pose estimationmodels during training to enforce the consistency betweenthe predicted3D poses and the ground-truth depth maps.However, these estimators rely on both RGB images and thepaired depth maps during training. In this study, we proposea conditional generative adversarial network (GAN) model,called Depth-image Guided GAN (DGGAN), to generate re-alistic depth maps conditioned on the input RGB image, anduse the synthesized depth maps to regularize the3D handpose estimation model, therefore eliminating the need forground-truth depth maps. Experimental results on multiplebenchmark datasets show that the synthesized depth mapsproduced by DGGAN are quite effective in regularizing thepose estimation model, yielding new state-of-the-art resultsin estimation accuracy, notably reducing the mean3D end-point errors (EPE) by4.7%,16.5%, and6.8%on the RHD,STB and MHP datasets, respectively.
University of Bonn（波恩大学）；National University of Singapore（新加坡国立大学）
Hand image synthesis and pose estimation from RGB images are both highly challenging tasks due to the large discrepancy between factors of variation ranging from image background content to camera viewpoint. To better analyze these factors of variation, we propose the use of disentangled representations and a disentangled variational autoencoder (dVAE) that allows for specific sampling and inference of these factors. The derived objective from the variational lower bound as well as the proposed training strategy are highly flexible, allowing us to handle cross-modal encoders and decoders as we…
We propose RANA, a relightable and articulated neural avatar for the photorealistic synthesis of humans under arbitrary viewpoints, body poses, and lighting. We only require a short video clip of the person to create the avatar and assume no knowledge about the lighting environment. We present a novel framework to model humans while disentangling their geometry, texture, and also lighting environment from monocular RGB videos. To simplify this otherwise ill-posed task we first estimate the coarse geometry and texture of the person via SMPL+D model fitting and then learn an articulated neural r…
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Imperial College London（伦敦帝国理工学院）；University of Oxford（牛津大学）；Technical University of Munich（慕尼黑工业大学）
For visual manipulation tasks, we aim to represent image content with semantically meaningful features. However, learning implicit representations from images often lacks interpretability, especially when attributes are intertwined. We focus on the challenging task of extracting disentangled 3D attributes only from 2D image data. Specifically, we focus on human appearance and learn implicit pose, shape and garment representations of dressed humans from RGB images. Our method learns an embedding with disentangled latent representations of these three image properties and enables meaningful re-a…
Carnegie Mellon University（卡内基梅隆大学）；Mila, University of Montreal（蒙特利尔大学米拉研究所）
We present neural architectures that disentangle RGB-D images into objects' shapes and styles and a map of the background scene, and explore their applications for few-shot 3D object detection and few-shot concept classification. Our networks incorporate architectural biases that reflect the image formation process, 3D geometry of the world scene, and shape-style interplay. They are trained end-to-end self-supervised by predicting views in static scenes, alongside a small number of 3D object boxes. Objects and scenes are represented in terms of 3D feature grids in the bottleneck of the network…
Vision-based robotic assembly is a crucial yet challenging task as the interaction with multiple objects requires high levels of precision. In this paper, we propose an integrated 6D robotic system to perceive, grasp, manipulate and assemble blocks with tight tolerances. Aiming to provide an off-the-shelf RGB-only solution, our system is built upon a monocular 6D object pose estimation network trained solely with synthetic images leveraging physically-based rendering. Subsequently, pose-guided 6D transformation along with collision-free assembly is proposed to construct any designed structure …
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）；Center for Excellence in Brain Science and Intelligence Technology（脑科学与智能技术卓越创新中心）
RGB-Infrared (IR) person re-identification is very challenging due to the large cross-modality variations between RGB and IR images. The key solution is to learn aligned features to the bridge RGB and IR modalities. However, due to the lack of correspondence labels between every pair of RGB and IR images, most methods try to alleviate the variations with set-level alignment by reducing the distance between the entire RGB and IR sets. However, this set-level alignment may lead to misalignment of some instances, which limits the performance for RGB-IR Re-ID. Different from existing methods, in t…
College of Engineering, Mathematics and Physical Sciences, University of Exeter（埃克塞特大学工程、数学与物理科学学院）；Institute of Computer Science (ICS), Foundation for Research and Technology - Hellas (FORTH)（希腊研究与技术基金会计算机科学研究所）
Human emotions analysis has been the focus of many studies, especially in the field of Affective Computing, and is important for many applications, e.g. human-computer intelligent interaction, stress analysis, interactive games, animations, etc. Solutions for automatic emotion analysis have also benefited from the development of deep learning approaches and the availability of vast amount of visual facial data on the internet. This paper proposes a novel method for human emotion recognition from a single RGB image. We construct a large-scale dataset of facial videos (\textbf{FaceVid}), rich in…
LIGM, École des Ponts, Univ Gustave Eiffel, CNRS（巴黎桥路学院、古斯塔夫·埃菲尔大学、法国国家科学研究中心信息、信号与系统实验室）；CIIRC CTU（捷克理工大学捷克信息学、机器人与控制论研究所）；ENS/Inria（巴黎高等师范学院/法国国家信息与自动化研究所）；Adobe Research（奥多比研究院）
We introduce FocalPose, a neural render-and-compare method for jointly estimating the camera-object 6D pose and camera focal length given a single RGB input image depicting a known object. The contributions of this work are twofold. First, we derive a focal length update rule that extends an existing state-of-the-art render-and-compare 6D pose estimator to address the joint estimation task. Second, we investigate several different loss functions for jointly estimating the object pose and focal length. We find that a combination of direct focal length regression with a reprojection loss disenta…
The University of Hong Kong（香港大学）；Brown University（布朗大学）；Microsoft Research Asia（微软亚洲研究院）；Centre for Garment Production Limited, Hong Kong（香港成衣生产中心有限公司）；SUSTech（南方科技大学）；Texas A&M University（德克萨斯农工大学）
Scalable 6D pose estimation for rigid objects from RGB images aims at handling multiple objects and generalizing to novel objects. Building on a well-known auto-encoding framework to cope with object symmetry and the lack of labeled training data, we achieve scalability by disentangling the latent representation of auto-encoder into shape and pose sub-spaces. The latent shape space models the similarity of different objects through contrastive metric learning, and the latent pose code is compared with canonical rotations for rotation retrieval. Because different object symmetries induce incons…
FocalPose++: Focal Length and Object Pose Estimation via Render and Compare
Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（布拉格捷克技术大学捷克信息学、机器人与控制研究所）；Reality Labs, Meta Platforms（元平台公司现实实验室）；Adobe Research（奥多比研究院）；LIGM, École des Ponts, UGE, CNRS（巴黎路桥学院LIGM实验室、巴黎东部大学、法国国家科学研究中心）
Cloth2Body: Generating 3D Human Body Mesh from 2D Clothing
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Technical University of Munich（慕尼黑工业大学）；ZMO AI Inc.（ZMO人工智能公司）；Nanyang Technological University（南洋理工大学）
