3D motion tracking is a critical task in many computer vision applications. Unsupervised markerless 3D motion tracking systems determine the most relevant object in the screen and then track it by continuously estimating its projection features (center and area) from the edge image and a point inside the relevant object projection (namely, inner point), until the tracking fails. Existing object projection feature estimation techniques are based on ray-casting from the inner point. These techniques present three main drawbacks: when the inner point is surrounded by edges, rays may not reach other relevant areas; as a consequence of that issue, the estimated features may greatly vary depending on the position of the inner point relative to the object projection; and finally, increasing the number of rays being casted and the ray-casting iterations (which would make the results more accurate and stable) increases the processing time to the point the tracking cannot be performed on the fly. In this paper, we analyze an intuitive filling-based object projection feature estimation technique that solves the aforementioned problems but is too sensitive to edge miscalculations. Then, we propose a less computing-intensive modification to that technique that would not be affected by the existing techniques issues and would be no more sensitive to edge miscalculations than ray-casting-based techniques.
The current state-of-the-art in monocular 3D human pose estimation is heavily influenced by weakly supervised methods. These allow 2D labels to be used to learn effective 3D human pose recovery either directly from images or via 2D-to-3D pose uplifting. In this paper we present a detailed analysis of the most commonly used simplified projection models, which relate the estimated 3D pose representation to 2D labels: normalized perspective and weak perspective projections. Specifically, we derive theoretical lower bound errors for those projection models under the commonly used mean per-joint po…
Victoria University of Wellington（惠灵顿维多利亚大学）；Adobe Research（Adobe研究院）
Optical flow computation is essential in the early stages of the video processing pipeline. This paper focuses on a less explored problem in this area, the 360$^\circ$ optical flow estimation using deep neural networks to support increasingly popular VR applications. To address the distortions of panoramic representations when applying convolutional neural networks, we propose a novel multi-projection fusion framework that fuses the optical flow predicted by the models trained using different projection methods. It learns to combine the complementary information in the optical flow results und…
We propose a method SPGNet for 3D human pose estimation that mixes multi-dimensional re-projection into supervised learning. In this method, the 2D-to-3D-lifting network predicts the global position and coordinates of the 3D human pose. Then, we re-project the estimated 3D pose back to the 2D key points along with spatial adjustments. The loss functions compare the estimated 3D pose with the 3D pose ground truth, and re-projected 2D pose with the input 2D pose. In addition, we propose a kinematic constraint to restrict the predicted target with constant human bone length. Based on the estimati…
Tsinghua University（清华大学）；Google（谷歌）；Technical University of Munich（慕尼黑工业大学）
Category-level object pose estimation aims to predict the 6D pose as well as the 3D metric size of arbitrary objects from a known set of categories. Recent methods harness shape prior adaptation to map the observed point cloud into the canonical space and apply Umeyama algorithm to recover the pose and size. However, their shape prior integration strategy boosts pose estimation indirectly, which leads to insufficient pose-sensitive feature extraction and slow inference speed. To tackle this problem, in this paper, we propose a novel geometry-guided Residual Object Bounding Box Projection netwo…
National Tsing Hua University（国立清华大学）；Phiar Technologies（Phiar科技公司）；National Yang Ming Chiao Tung University（国立阳明交通大学）
Due to the rise of spherical cameras, monocular 360 depth estimation becomes an important technique for many applications (e.g., autonomous systems). Thus, state-of-the-art frameworks for monocular 360 depth estimation such as bi-projection fusion in BiFuse are proposed. To train such a framework, a large number of panoramas along with the corresponding depth ground truths captured by laser sensors are required, which highly increases the cost of data collection. Moreover, since such a data collection procedure is time-consuming, the scalability of extending these methods to different scenes b…
Tampere University（坦佩雷大学）；Queen Mary University of London（伦敦玛丽女王大学）
Human-robot object handover is a key skill for the future of human-robot collaboration. CORSMAL 2020 Challenge focuses on the perception part of this problem: the robot needs to estimate the filling mass of a container held by a human. Although there are powerful methods in image processing and audio processing individually, answering such a problem requires processing data from multiple sensors together. The appearance of the container, the sound of the filling, and the depth data provide essential information. We propose a multi-modal method to predict three key indicators of the filling mas…
Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）
Real-time 3D human pose estimation is crucial for human-computer interaction. It is cheap and practical to estimate 3D human pose only from monocular video. However, recent bone splicing based 3D human pose estimation method brings about the problem of cumulative error. In this paper, the concept of virtual bones is proposed to solve such a challenge. The virtual bones are imaginary bones between non-adjacent joints. They do not exist in reality, but they bring new loop constraints for the estimation of 3D human joints. The proposed network in this paper predicts real bones and virtual bones, …
Institute of Computer Graphics and Vision, Graz University of Technology, Austria（奥地利格拉茨工业大学计算机图形与视觉研究所）；Laboratoire Bordelais de Recherche en Informatique, University of Bordeaux, France（法国波尔多大学波尔多信息学实验室）
We present a joint 3D pose and focal length estimation approach for object categories in the wild. In contrast to previous methods that predict 3D poses independently of the focal length or assume a constant focal length, we explicitly estimate and integrate the focal length into the 3D pose estimation. For this purpose, we combine deep learning techniques and geometric algorithms in a two-stage approach: First, we estimate an initial focal length and establish 2D-3D correspondences from a single RGB image using a deep network. Second, we recover 3D poses and refine the focal length by minimiz…
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）
360° images are widely available over the last few years. This paper proposes a new technique for single 360° image depth prediction under open environments. Depth prediction from a 360° single image is not easy for two reasons. One is the limitation of supervision datasets - the currently available dataset is limited to indoor scenes. The other is the problems caused by Equirectangular Projection Format (ERP), commonly used for 360° images, that are coordinate and distortion. There is only one method existing that uses cube map projection to produce six perspective images and apply self-super…
Adaptive Radial Projection on Fourier Magnitude Spectrum for Document Image Skew Estimation
Cinnamon AI - Department of Research Floor 10th, No. 36 Hoang Cau street, O Cho Dua Ward, Dong Da District, Hanoi City, Viet Nam. Faculty of Computer Science \& Engineering, Ho Chi Minh City-University of Technology (HCMUT), 268 Ly Thuong Kiet Street, District 10, Ho Chi Minh City, Vietnam Vietnam National University Ho Chi Minh City, Linh Trung Ward, Thu Duc District, Ho Chi Minh City, Vietnam.
Institute for Information Transmission Problems – IITP RAS（俄罗斯科学院信息传输问题研究所）；Moscow Institute of Physics and Technology (National Research University)（莫斯科物理技术学院（国立研究型大学））
This paper proposes a RANSAC-based algorithm for determining the axial rotation angle of an object from a pair of its tomographic projections. An equation is derived for calculating the rotation angle using one correct keypoints correspondence of two tomographic projections. The proposed algorithm consists of the following steps: keypoints detection and matching, rotation angle estimation for each correspondence, outliers filtering with the RANSAC algorithm, finally, calculation of the desired angle by minimizing the re-projection error from the remaining correspondences. To validate the propo…
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）
While the voxel-based methods have achieved promising results for multi-person 3D pose estimation from multi-cameras, they suffer from heavy computation burdens, especially for large scenes. We present Faster VoxelPose to address the challenge by re-projecting the feature volume to the three two-dimensional coordinate planes and estimating X, Y, Z coordinates from them separately. To that end, we first localize each person by a 3D bounding box by estimating a 2D box and its height based on the volume features projected to the xy-plane and z-axis, respectively. Then for each person, we estimate…
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
In recent years, estimating the 6D pose of object instances with convolutional neural network (CNN) has received considerable attention. Depending on whether intermediate cues are used, the relevant literature can be roughly divided into two broad categories: direct methods and two stage pipelines. For the latter, intermediate cues, such as 3D object coordinates, semantic keypoints, or virtual control points instead of pose parameters are regressed by CNN in the first stage. Object pose can then be solved by correspondence constraints constructed with these intermediate cues. In this paper, we…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Ant Financial（蚂蚁金服）；Alibaba Group US（阿里巴巴集团美国）
Gradient estimation and vector space projection have been studied as two distinct topics. We aim to bridge the gap between the two by investigating how to efficiently estimate gradient based on a projected low-dimensional space. We first provide lower and upper bounds for gradient estimation under both linear and nonlinear projections, and outline checkable sufficient conditions under which one is better than the other. Moreover, we analyze the query complexity for the projection-based gradient estimation and present a sufficient condition for query-efficient estimators. Built upon our theoret…
Signal Processing Friedrich-Alexander University Erlangen-N\"urnberg (FAU), Cauerstr. 7, 91058 Erlangen, Germany
Capturing large fields of view with only one camera is an important aspect in surveillance and automotive applications, but the wide-angle fisheye imagery thus obtained exhibits very special characteristics that may not be very well suited for typical image and video processing methods such as motion estimation. This paper introduces a motion estimation method that adapts to the typical radial characteristics of fisheye video sequences by making use of an equisolid re-projection after moving part of the motion vector search into the perspective domain via a corresponding back-projection. By co…
Megvii Technology（旷视科技）；University of Electronic Science and Technology of China（电子科技大学）
In this paper, we introduce a new framework for unsupervised deep homography estimation. Our contributions are 3 folds. First, unlike previous methods that regress 4 offsets for a homography, we propose a homography flow representation, which can be estimated by a weighted sum of 8 pre-defined homography flow bases. Second, considering a homography contains 8 Degree-of-Freedoms (DOFs) that is much less than the rank of the network features, we propose a Low Rank Representation (LRR) block that reduces the feature rank, so that features corresponding to the dominant motions are retained while o…
Images of both rotating celestial bodies (e.g., asteroids) and spheroidal planets with banded atmospheres (e.g., Jupiter) can contain features that are well-modeled as a circle of latitude (CoL). The projections of these CoLs appear as ellipses in images collected by cameras or telescopes onboard exploration spacecraft. This work shows how CoL projections may be used to determine the pole orientation and covariance for a spinning asteroid. In the case of a known planet modeled as an oblate spheroid, it is shown how similar CoL projections may be used for spacecraft localization. These methods …
SenseTime Research（商汤科技研究院）；University of Chinese Academy of Sciences（中国科学院大学）
As RGB-D sensors become more affordable, using RGB-D images to obtain high-accuracy 6D pose estimation results becomes a better option. State-of-the-art approaches typically use different backbones to extract features for RGB and depth images. They use a 2D CNN for RGB images and a per-pixel point cloud network for depth data, as well as a fusion network for feature fusion. We find that the essential reason for using two independent backbones is the "projection breakdown" problem. In the depth image plane, the projected 3D structure of the physical world is preserved by the 1D depth value and …
University College London（伦敦大学学院）；Queen Mary University of London（伦敦玛丽女王大学）；University of Birmingham（伯明翰大学）
We present methods to estimate the physical properties of household containers and their fillings manipulated by humans. We use a lightweight, pre-trained convolutional neural network with coordinate attention as a backbone model of the pipelines to accurately locate the object of interest and estimate the physical properties in the CORSMAL Containers Manipulation (CCM) dataset. We address the filling type classification with audio data and then combine this information from audio with video modalities to address the filling level classification. For the container capacity, dimension, and mass…
Convolutional neural networks (CNNs) have yielded the excellent performance in a variety of computer vision tasks, where CNNs typically adopt a similar structure consisting of convolution layers, pooling layers and fully connected layers. In this paper, we propose to apply a novel method, namely Hybrid Orthogonal Projection and Estimation (HOPE), to CNNs in order to introduce orthogonality into the CNN structure. The HOPE model can be viewed as a hybrid model to combine feature extraction using orthogonal linear projection with mixture models. It is an effective model to extract useful informa…
Bone mineral density estimation from a plain X-ray image by learning decomposition into projections of bone-segmented computed tomography
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；CentraleSupélec（巴黎中央理工-高等电力学院）；Université Paris-Saclay（巴黎萨克雷大学）；Inria（法国国家信息与自动化研究所）；Osaka University Graduate School of Medicine（大阪大学医学研究生院）；Ehime University Graduate School of Medicine（爱媛大学医学研究生院）
Tencent AI Lab（腾讯AI实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）；UCLA（加州大学洛杉矶分校）
Many man-made objects have intrinsic symmetries and Manhattan structure. By assuming an orthographic projection model, this paper addresses the estimation of 3D structures and camera projection using symmetry and/or Manhattan structure cues, which occur when the input is single- or multiple-image from the same category, e.g., multiple different cars. Specifically, analysis on the single image case implies that Manhattan alone is sufficient to recover the camera projection, and then the 3D structure can be reconstructed uniquely exploiting symmetry. However, Manhattan structure can be difficult…
Leibniz Universität Hannover（汉诺威莱布尼茨大学）；Institute of Mechatronic Systems（机电系统研究所）
In microsurgery, lasers have emerged as precise tools for bone ablation. A challenge is automatic control of laser bone ablation with 4D optical coherence tomography (OCT). OCT as high resolution imaging modality provides volumetric images of tissue and foresees information of bone position and orientation (pose) as well as thickness. However, existing approaches for OCT based laser ablation control rely on external tracking systems or invasively ablated artificial landmarks for tracking the pose of the OCT probe relative to the tissue. This can be superseded by estimating the scene flow cause…
