Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction
Imperial College London（帝国理工学院）；Niantic, Inc.（Niantic公司）；Graz Uni. of Technology（格拉茨工业大学）；Rokid Corp. Ltd.（Rokid公司）；HUST（华中科技大学）；Megvii Research Nanjing（旷视南京研究院）；SUNY Buffalo（纽约州立大学布法罗分校）；BUPT（北京邮电大学）；Uni. of West Bohemia（西波希米亚大学）；Fudan Uni.（复旦大学）；TUM（慕尼黑工业大学）；Uni. of Bonn（波恩大学）；NUS（新加坡国立大学）；Harbin Inst. of Technology（哈尔滨工业大学）；CUHK（香港中文大学）；ETH Zurich（苏黎世联邦理工学院）；NVIDIA Research（英伟达研究院）；NAVER LABS Europe（NAVER欧洲实验室）；ENPC ParisTech（巴黎高科路桥学校）；UNIST（蔚山科学技术院）；KAIST（韩国科学技术院）
We study how well different types of approaches generalise in the task of 3D hand pose estimation under single hand scenarios and hand-object interaction. We show that the accuracy of state-of-the-art methods can drop, and that they fail mostly on poses absent from the training set. Unfortunately, since the space of hand poses is highly dimensional, it is inherently not feasible to cover the whole space densely, despite recent efforts in collecting large-scale training datasets. This sampling problem is even more severe when hands are interacting with objects and/or inputs are RGB rather than depth images, as RGB images also vary with lighting conditions and colors. To address these issues, we designed a public challenge (HANDS'19) to evaluate the abilities of current 3D hand pose estimators (HPEs) to interpolate and extrapolate the poses of a training set. More exactly, HANDS'19 is designed (a) to evaluate the influence of both depth and color modalities on 3D hand pose estimation, under the presence or absence of objects; (b) to assess the generalisation abilities w.r.t. four main axes: shapes, articulations, viewpoints, and objects; (c) to explore the use of a synthetic hand model to fill the gaps of current datasets. Through the challenge, the overall accuracy has dramatically improved over the baseline, especially on extrapolation tasks, from 27mm to 13mm mean joint error. Our analyses highlight the impacts of: Data pre-processing, ensemble approaches, the use of a parametric 3D hand model (MANO), and different HPE methods/backbones.
Tsinghua University（清华大学）；University College London（伦敦大学学院）；Beijing Huajie IMI Technology Co., Ltd.（北京华捷艾米科技有限公司）
In human-computer interaction, it is important to accurately estimate the hand pose especially fingertips. However, traditional approaches for fingertip localization mainly rely on depth images and thus suffer considerably from the noise and missing values. Instead of depth images, stereo images can also provide 3D information of hands and promote 3D hand pose estimation. There are nevertheless limitations on the dataset size, global viewpoints, hand articulations and hand shapes in the publicly available stereo-based hand pose datasets. To mitigate these limitations and promote further resear…
We present the 2017 Hands in the Million Challenge, a public competition designed for the evaluation of the task of 3D hand pose estimation. The goal of this challenge is to assess how far is the state of the art in terms of solving the problem of 3D hand pose estimation as well as detect major failure and strength modes of both systems and evaluation metrics that can help to identify future research directions. The challenge follows up the recent publication of BigHand2.2M and First-Person Hand Action datasets, which have been designed to exhaustively cover multiple hand, viewpoint, hand arti…
Joint Multi-Condition Representation Modelling via Matrix Factorisation for Visual Place Recognition
School of Electronics and Computer Science, University of Southampton（苏塞克斯大学电子与计算机科学学院）；Department of Computer Science and Engineering, Hanyang University（翰阳大学计算机科学与工程系）；QUT Centre for Robotics, School of Electrical Engineering and Robotics（昆士兰大学机器人中心、电气工程与机器人学院）；School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）
In this article we consider a version of the challenging problem of learning from datasets whose size is too limited to allow generalisation beyond the training set. To address the challenge we propose to use a transfer learning approach whereby the model is first trained on a synthetic dataset replicating features of the original objects. In this study the objects were smartphone photographs of near-complete Roman terra sigillata pottery vessels from the collection of the Museum of London. Taking the replicated features from published profile drawings of pottery forms allowed the integration …
Learning and predicting the pose parameters of a 3D hand model given an image, such as locations of hand joints, is challenging due to large viewpoint changes and articulations, and severe self-occlusions exhibited particularly in egocentric views. Both feature learning and prediction modeling have been investigated to tackle the problem. Though effective, most existing discriminative methods yield a single deterministic estimation of target poses. Due to their single-value mapping intrinsic, they fail to adequately handle self-occlusion problems, where occluded joints present multiple modes. …
Discriminative methods often generate hand poses kinematically implausible, then generative methods are used to correct (or verify) these results in a hybrid method. Estimating 3D hand pose in a hierarchy, where the high-dimensional output space is decomposed into smaller ones, has been shown effective. Existing hierarchical methods mainly focus on the decomposition of the output space while the input space remains almost the same along the hierarchy. In this paper, a hybrid hand pose estimation method is proposed by applying the kinematic hierarchy strategy to the input space (as well as the …
University of Surrey（萨里大学）；Vicon Motion Systems Ltd（维康运动系统有限公司）
We present a novel method to improve the accuracy of the 3D reconstruction of clothed human shape from a single image. Recent work has introduced volumetric, implicit and model-based shape learning frameworks for reconstruction of objects and people from one or more images. However, the accuracy and completeness for reconstruction of clothed people is limited due to the large variation in shape resulting from clothing, hair, body size, pose and camera viewpoint. This paper introduces two advances to overcome this limitation: firstly a new synthetic dataset of realistic clothed people, 3DVH; an…
University of Zagreb, Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）
In this paper we propose a novel approach to generate a synthetic aerial dataset for application in UAV monitoring. We propose to accentuate shape-based object representation by applying texture randomization. A diverse dataset with photorealism in all parameters such as shape, pose, lighting, scale, viewpoint, etc. except for atypical textures is created in a 3D modelling software Blender. Our approach specifically targets two conditions in aerial images where texture of objects is difficult to detect, namely challenging illumination and objects occupying only a small portion of the image. Ex…
