In bin-picking scenarios, multiple instances of an object of interest are stacked in a pile randomly, and hence, the instances are inherently subjected to the challenges: severe occlusion, clutter, and similar-looking distractors. Most existing methods are, however, for single isolated object instances, while some recent methods tackle crowd scenarios as post-refinement which accounts multiple object relations. In this paper, we address recovering 6D poses of multiple instances in bin-picking scenarios in depth modality by multi-task learning in deep neural networks. Our architecture jointly learns multiple sub-tasks: 2D detection, depth, and 3D pose estimation of individual objects; and joint registration of multiple objects. For training data generation, depth images of physically plausible object pose configurations are generated by a 3D object model in a physics simulation, which yields diverse occlusion patterns to learn. We adopt a state-of-the-art object detector, and 2D offsets are further estimated via a network to refine misaligned 2D detections. The depth and 3D pose estimator is designed to generate multiple hypotheses per detection. This allows the joint registration network to learn occlusion patterns and remove physically implausible pose hypotheses. We apply our architecture on both synthetic (our own and Sileane dataset) and real (a public Bin-Picking dataset) data, showing that it significantly outperforms state-of-the-art methods by 15-31% in average precision.
The Johns Hopkins University（约翰斯·霍普金斯大学）；National University of Singapore（新加坡国立大学）
Epipolar constraints are at the core of feature matching and depth estimation in current multi-person multi-camera 3D human pose estimation methods. Despite the satisfactory performance of this formulation in sparser crowd scenes, its effectiveness is frequently challenged under denser crowd circumstances mainly due to two sources of ambiguity. The first is the mismatch of human joints resulting from the simple cues provided by the Euclidean distances between joints and epipolar lines. The second is the lack of robustness from the naive formulation of the problem as a least squares minimizatio…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Fraunhofer Institute for Optronics, System Technologies and Image Exploitation IOSB（弗劳恩霍夫光电、系统技术与图像利用研究所IOSB）
Human pose estimation has recently made significant progress with the adoption of deep convolutional neural networks. Its many applications have attracted tremendous interest in recent years. However, many practical applications require pose estimation for human crowds, which still is a rarely addressed problem. In this work, we explore methods to optimize pose estimation for human crowds, focusing on challenges introduced with dense crowds, such as occlusions, people in close proximity to each other, and partial visibility of people. In order to address these challenges, we evaluate three asp…
National University of Singapore（新加坡国立大学）；YITU Technology（依图科技）
Video-based human pose estimation in crowded scenes is a challenging problem due to occlusion, motion blur, scale variation and viewpoint change, etc. Prior approaches always fail to deal with this problem because of (1) lacking of usage of temporal information; (2) lacking of training data in crowded scenes. In this paper, we focus on improving human pose estimation in videos of crowded scenes from the perspectives of exploiting temporal context and collecting new data. In particular, we first follow the top-down strategy to detect persons and perform single-person pose estimation for each fr…
While much progress has been made on the task of 3D point cloud registration, there still exists no learning-based method able to estimate the 6D pose of an object observed by a 2.5D sensor in a scene. The challenges of this scenario include the fact that most measurements are outliers depicting the object's surrounding context, and the mismatch between the complete 3D object model and its self-occluded observations. We introduce the first deep learning framework capable of effectively handling this scenario. Our method consists of an instance segmentation module followed by a pose estimation …
SAM-pose2seg: Pose-Guided Human Instance Segmentation in Crowds
Visual Recognition Group（视觉识别组）；Department of Cybernetics（cybernetics系）；Faculty of Electrical Engineering（电气工程系）；Czech Technical University in Prague（布拉格捷克技术大学）
A key assumption of top-down human pose estimation approaches is their expectation of having a single person/instance present in the input bounding box. This often leads to failures in crowded scenes with occlusions. We propose a novel solution to overcome the limitations of this fundamental assumption. Our Multi-Instance Pose Network (MIPNet) allows for predicting multiple 2D pose instances within a given bounding box. We introduce a Multi-Instance Modulation Block (MIMB) that can adaptively modulate channel-wise feature responses for each instance and is parameter efficient. We demonstrate t…
Toga Networks（Toga网络公司）；Huawei（华为）；Technion（以色列理工学院）
As scene changes with time map descriptors become outdated, affecting VPS localization accuracy. In this work, we propose an approach to detect structural and texture scene changes to be followed by map update. In our method - map includes 3D points with descriptors generated either via LiDAR or SFM. Common approaches suffer from shortcomings: 1) Direct comparison of the two point-clouds for change detection is slow due to the need to build new point-cloud every time we want to compare; 2) Image based comparison requires to keep the map images adding substantial storage overhead. To circumvent…
Regression-based Pelvic Pose Initialization for Fast and Robust 2D/3D Pelvis Registration
Department of Computer Science（计算机科学系）；Vanderbilt University（范德比大学）；Vanderbilt Institute of Surgery and Engineering（范德比手术与工程研究所）；Vanderbilt Lab for Immersive AI Translation（范德比沉浸式AI翻译实验室）；Department of Orthopaedic Surgery（骨科手术系）；Vanderbilt University Medical Center（范德比大学医学中心）
Deep-PE: A Learning-Based Pose Evaluator for Point Cloud Registration
Shandong University（山东大学）；Shandong University of Science and Technology（山东科技大学）；National Deep Sea Center（国家深海中心）；Qingdao University of Science and Technology（青岛科技大学）；Texas A&M University（德克萨斯农工大学）
Multi-person pose estimation is fundamental to many computer vision tasks and has made significant progress in recent years. However, few previous methods explored the problem of pose estimation in crowded scenes while it remains challenging and inevitable in many scenarios. Moreover, current benchmarks cannot provide an appropriate evaluation for such cases. In this paper, we propose a novel and efficient method to tackle the problem of pose estimation in the crowd and a new dataset to better evaluate algorithms. Our model consists of two key components: joint-candidate single person pose est…
Seoul National University（首尔大学）；Harvard University（哈佛大学）；Seoul National University School of Dentistry（首尔大学牙医学院）；Soong-sil University（崇实大学）
Computerized registration between maxillofacial cone-beam computed tomography (CT) images and a scanned dental model is an essential prerequisite in surgical planning for dental implants or orthognathic surgery. We propose a novel method that performs fully automatic registration between a cone-beam CT image and an optically scanned model. To build a robust and automatic initial registration method, our method applies deep-pose regression neural networks in a reduced domain (i.e., 2-dimensional image). Subsequently, fine registration is performed via optimal clusters. Majority voting system ac…
Northwestern Polytechnical University（西北工业大学）；Baidu Research（百度研究院）；National Engineering Laboratory of Deep Learning Technology and Application（深度学习技术及应用国家工程实验室）
3D point cloud registration in remote sensing field has been greatly advanced by deep learning based methods, where the rigid transformation is either directly regressed from the two point clouds (correspondences-free approaches) or computed from the learned correspondences (correspondences-based approaches). Existing correspondences-free methods generally learn the holistic representation of the entire point cloud, which is fragile for partial and noisy point clouds. In this paper, we propose a correspondences-free unsupervised point cloud registration (UPCR) method from the representation se…
Point cloud registration is a common step in many 3D computer vision tasks such as object pose estimation, where a 3D model is aligned to an observation. Classical registration methods generalize well to novel domains but fail when given a noisy observation or a bad initialization. Learning-based methods, in contrast, are more robust but lack in generalization capacity. We propose to consider iterative point cloud registration as a reinforcement learning task and, to this end, present a novel registration agent (ReAgent). We employ imitation learning to initialize its discrete registration pol…
University of Utah（犹他大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
Popular 3D scan registration projects, such as Stanford digital Michelangelo or KinectFusion, exploit the high-resolution sensor data for scan alignment. It is particularly challenging to solve the registration of sparse 3D scans in the absence of RGB components. In this case, we can not establish point correspondences since the same 3D point cannot be captured in two successive scans. In contrast to correspondence based methods, we take a different viewpoint and formulate the sparse 3D registration problem based on the constraints from the intersection of line segments from adjacent scans. We…
Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Northeastern University（东北大学）
With an aim to increase the capture range and accelerate the performance of state-of-the-art inter-subject and subject-to-template 3D registration, we propose deep learning-based methods that are trained to find the 3D position of arbitrarily oriented subjects or anatomy based on slices or volumes of medical images. For this, we propose regression CNNs that learn to predict the angle-axis representation of 3D rotations and translations using image features. We use and compare mean square error and geodesic loss to train regression CNNs for 3D pose estimation used in two different scenarios: sl…
LayersReg: A Layer-by-Layer Progressive Regressor for Reliable Intraoperative 3D/2D Registration
School of Biological Science and Medical Engineering（生物科学与医学工程学院）；Southeast University（东南大学）；School of Engineering, College of Engineering and Physical Sciences（工程学院，工程与物理科学学院）；University of Birmingham（伯明翰大学）；State Key Laboratory of Digital Medical Engineering（数字医学工程国家重点实验室）
In this work, we tackle the task of estimating the 6D pose of an object from point cloud data. While recent learning-based approaches to addressing this task have shown great success on synthetic datasets, we have observed them to fail in the presence of real-world data. We thus analyze the causes of these failures, which we trace back to the difference between the feature distributions of the source and target point clouds, and the sensitivity of the widely-used SVD-based loss function to the range of rotation between the two point clouds. We address the first challenge by introducing a new n…
National University of Singapore（新加坡国立大学）；YITU Technology（依图科技）
This paper presents our solution to ACM MM challenge: Large-scale Human-centric Video Analysis in Complex Events\cite{lin2020human}; specifically, here we focus on Track3: Crowd Pose Tracking in Complex Events. Remarkable progress has been made in multi-pose training in recent years. However, how to track the human pose in crowded and complex environments has not been well addressed. We formulate the problem as several subproblems to be solved. First, we use a multi-object tracking method to assign human ID to each bounding box generated by the detection model. After that, a pose is generated …
MemReg: Streaming Outdoor LiDAR Point Cloud Registration with Hybrid Memory Buffers
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）；University of Twente（特文特大学）；University of Cambridge（剑桥大学）；University of Bath（巴斯大学）；Phigent Robotics（飞劲机器人公司）
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems Tübingen（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）
Registering point clouds of dressed humans to parametric human models is a challenging task in computer vision. Traditional approaches often rely on heavily engineered pipelines that require accurate manual initialization of human poses and tedious post-processing. More recently, learning-based methods are proposed in hope to automate this process. We observe that pose initialization is key to accurate registration but existing methods often fail to provide accurate pose initialization. One major obstacle is that, regressing joint rotations from point clouds or images of humans is still very c…
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Hong Kong University of Science and Technology（香港科技大学）；Technical University of Munich（慕尼黑工业大学）
Direct image-to-image alignment that relies on the optimization of photometric error metrics suffers from limited convergence range and sensitivity to lighting conditions. Deep learning approaches has been applied to address this problem by learning better feature representations using convolutional neural networks, yet still require a good initialization. In this paper, we demonstrate that the inaccurate numerical Jacobian limits the convergence range which could be improved greatly using learned approaches. Based on this observation, we propose a novel end-to-end network, RegNet, to learn th…
CSS: Overcoming Pose and Scene Challenges in Crowd-Sourced 3D Gaussian Splatting
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；China Unicom Smart City Research Institute（中国联通智慧城市研究院）
Pose registration is critical in vision and robotics. This paper focuses on the challenging task of initialization-free pose registration up to 7DoF for homogeneous and heterogeneous measurements. While recent learning-based methods show promise using differentiable solvers, they either rely on heuristically defined correspondences or are prone to local minima. We present a differentiable phase correlation (DPC) solver that is globally convergent and correspondence-free. When combined with simple feature extraction networks, our general framework DPCN++ allows for versatile pose registration w…
University of Rochester（罗切斯特大学）；University of Maryland, College Park（马里兰大学帕克分校）；Rutgers University（罗格斯大学）；Z2W Corporation（Z2W公司）；Chinese Academy of Sciences（中国科学院）；Peng Cheng Laboratory（鹏城实验室）
We propose to tackle the problem of multiview 2D/3D rigid registration for intervention via a Point-Of-Interest Network for Tracking and Triangulation ($\text{POINT}^2$). $\text{POINT}^2$ learns to establish 2D point-to-point correspondences between the pre- and intra-intervention images by tracking a set of random POIs. The 3D pose of the pre-intervention volume is then estimated through a triangulation layer. In $\text{POINT}^2$, the unified framework of the POI tracker and the triangulation layer enables learning informative 2D features and estimating 3D pose jointly. In contrast to existin…
We propose a minimal solution for the similarity registration (rigid pose and scale) between two sets of 3D lines, and also between a set of co-planar points and a set of 3D lines. The first problem is solved up to 8 discrete solutions with a minimum of 2 line-line correspondences, while the second is solved up to 4 discrete solutions using 4 point-line correspondences. We use these algorithms to perform the extrinsic calibration between a pose tracking sensor and a 2D/3D ultrasound (US) curvilinear probe using a tracked needle as calibration target. The needle is tracked as a 3D line, and is …
IMATI-CNR（意大利国家研究委员会应用数学与信息技术研究所）；University of Maryland（马里兰大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
There has been significant work on learning realistic, articulated, 3D models of the human body. In contrast, there are few such models of animals, despite many applications. The main challenge is that animals are much less cooperative than humans. The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals. Consequently, we learn our model from a small set of 3D scans of toy figurines in arbitrary poses. We employ a novel part-based shape model to compute an initial registration to the scans. We then normalize their pose…
Enhanced Landmark Detection Model in Pelvic Fluoroscopy using 2D/3D Registration Loss
Department of Computer Science, Vanderbilt University（计算机科学系，范德堡大学）；Vanderbilt Institute for Surgery and Engineering（范德堡手术与工程研究院）；Department of Mathematics, University of California-Los Angeles（数学系，加州大学洛杉矶分校）；Vanderbilt Lab for Immersive AI Translation（范德堡沉浸式AI翻译实验室）；Department of Orthopedic Surgery, Vanderbilt University Medical Center（骨科手术系，范德堡大学医学中心）
C3VDReg: A Benchmark for Local-to-Local Colonoscopic Registration toward Anatomical Localization
UCL Hawkes Institute（UCL哈维斯研究所）；University College London, University of London（伦敦大学学院, 伦敦大学）；Division of Informatics, Imaging and Data Sciences（信息学、成像与数据科学部门）；University of Manchester（曼彻斯特大学）
Depth Augmented and FE Free 3D/2D Liver Registration for Laparoscopic Liver AR
UCL Hawkes Institute, University College London, London WC1E 6BT, UK（伦敦大学学院UCL哈维斯研究所）；Division of Surgery and Interventional Science, University College London, London WC1E 6BT, UK（伦敦大学学院UCL外科与介入科学系）；Unit for Lifelong Health and Ageing at UCL, University College London, London WC1E 7HB, UK（伦敦大学学院UCL终身健康与老龄化单位）；Medtronic plc., London, UK（伦敦梅脱利克公司）
National Taiwan University（国立台湾大学）；Mobile Drive Technology（移动驱动科技公司）
Point cloud registration is a crucial problem in computer vision and robotics. Existing methods either rely on matching local geometric features, which are sensitive to the pose differences, or leverage global shapes, which leads to inconsistency when facing distribution variances such as partial overlapping. Combining the advantages of both types of methods, we adopt a coarse-to-fine pipeline that concurrently handles both issues. We first reduce the pose differences between input point clouds by aligning global features; then we match the local features to further refine the inaccurate align…
Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Fully automatic X-ray to CT registration requires a solid initialization to provide an initial alignment within the capture range of existing intensity-based registrations. This work adresses that need by providing a novel automatic initialization, which enables end to end registration. First, a neural network is trained once to detect a set of anatomical landmarks on simulated X-rays. A domain randomization scheme is proposed to enable the network to overcome the challenge of being trained purely on simulated data and run inference on real Xrays. Then, for each patient CT, a patient-specific …
