Keypoint detection is an essential building block for many robotic applications like motion capture and pose estimation. Historically, keypoints are detected using uniquely engineered markers such as checkerboards or fiducials. More recently, deep learning methods have been explored as they have the ability to detect user-defined keypoints in a marker-less manner. However, different manually selected keypoints can have uneven performance when it comes to detection and localization. An example of this can be found on symmetric robotic tools where DNN detectors cannot solve the correspondence problem correctly. In this work, we propose a new and autonomous way to define the keypoint locations that overcomes these challenges. The approach involves finding the optimal set of keypoints on robotic manipulators for robust visual detection and localization. Using a robotic simulator as a medium, our algorithm utilizes synthetic data for DNN training, and the proposed algorithm is used to optimize the selection of keypoints through an iterative approach. The results show that when using the optimized keypoints, the detection performance of the DNNs improved significantly. We further use the optimized keypoints for real robotic applications by using domain randomization to bridge the reality gap between the simulator and the physical world. The physical world experiments show how the proposed method can be applied to the wide-breadth of robotic applications that require visual feedback, such as camera-to-robot calibration, robotic tool tracking, and end-effector pose estimation.
In this paper, we introduce a rotational primitive prediction based 6D object pose estimation using a single image as an input. We solve for the 6D object pose of a known object relative to the camera using a single image with occlusion. Many recent state-of-the-art (SOTA) two-step approaches have exploited image keypoints extraction followed by PnP regression for pose estimation. Instead of relying on bounding box or keypoints on the object, we propose to learn orientation-induced primitive so as to achieve the pose estimation accuracy regardless of the object size. We leverage a Variational …
We propose a fast and accurate method of 6D object pose estimation for bin-picking of mechanical parts by a robot manipulator. We extend the single-shot approach to stereo vision by application of attention architecture. Our convolutional neural network model regresses to object locations and rotations from either a left image or a right image without depth information. Then, a stereo feature matching module, designated as Stereo Grid Attention, generates stereo grid matching maps. The important point of our method is only to calculate disparity of the objects found by the attention from stere…
The two-stage object pose estimation paradigm first detects semantic keypoints on the image and then estimates the 6D pose by minimizing reprojection errors. Despite performing well on standard benchmarks, existing techniques offer no provable guarantees on the quality and uncertainty of the estimation. In this paper, we inject two fundamental changes, namely conformal keypoint detection and geometric uncertainty propagation, into the two-stage paradigm and propose the first pose estimator that endows an estimation with provable and computable worst-case error bounds. On one hand, conformal ke…
The task of 6D object pose estimation from RGB images is an important requirement for autonomous service robots to be able to interact with the real world. In this work, we present a two-step pipeline for estimating the 6 DoF translation and orientation of known objects. Keypoints and Part Affinity Fields (PAFs) are predicted from the input image adopting the OpenPose CNN architecture from human pose estimation. Object poses are then calculated from 2D-3D correspondences between detected and model keypoints via the PnP-RANSAC algorithm. The proposed approach is evaluated on the YCB-Video datas…
Keypoint-based Dynamic Object 6-DoF Pose Tracking via Event Camera
ShanghaiTech University（上海科技大学）；ShanghaiTech Automation and Robotics (STAR) Center（上海科技大学自动化与机器人中心）；School of Information Science and Technology（信息科学与技术学院）；WLSA Shanghai Academy（WLSA上海研究院）
Hong Kong University of Science and Technology（香港科技大学）；Megvii Inc.（旷视科技）；Kuaishou Technology（快手科技）
In this work, we present a novel data-driven method for robust 6DoF object pose estimation from a single RGBD image. Unlike previous methods that directly regressing pose parameters, we tackle this challenging task with a keypoint-based approach. Specifically, we propose a deep Hough voting network to detect 3D keypoints of objects and then estimate the 6D pose parameters within a least-squares fitting manner. Our method is a natural extension of 2D-keypoint approaches that successfully work on RGB based 6DoF estimation. It allows us to fully utilize the geometric constraint of rigid objects w…
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data. Our method tracks in real-time novel object instances of known object categories such as bowls, laptops, and mugs. 6-PACK learns to compactly represent an object by a handful of 3D keypoints, based on which the interframe motion of an object instance can be estimated through keypoint matching. These keypoints are learned end-to-end without manual supervision in order to be most effective for tracking. Our experiments show that our method substantially outperforms existing methods on the NOCS ca…
University of Pennsylvania（宾夕法尼亚大学）；Ryerson University（瑞尔森大学）
This paper presents a novel approach to estimating the continuous six degree of freedom (6-DoF) pose (3D translation and rotation) of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior work, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training image data. Furthermore, the approach can be applied to instance- and class-based pose recovery. Empirically, we show that the proposed approach can a…
DKPMV: Dense Keypoints Fusion from Multi-View RGB Frames for 6D Pose Estimation of Textureless Objects
College of Aerospace Science and Engineering, National University of Defense Technology（航空航天科学与工程学院，国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（图像测量与视觉导航省级重点实验室）
MK-Pose: Category-Level Object Pose Estimation via Multimodal-Based Keypoint Learning
Institute of Robotics and Automatic Information System（机器人与自动信息系统研究所）；Tianjin Key Laboratory of Intelligent Robotics（天津智能机器人重点实验室）；TBI center（TBI中心）；Nankai University（南开大学）
NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）
Prior work on 6-DoF object pose estimation has largely focused on instance-level processing, in which a textured CAD model is available for each object being detected. Category-level 6-DoF pose estimation represents an important step toward developing robotic vision systems that operate in unstructured, real-world scenarios. In this work, we propose a single-stage, keypoint-based approach for category-level object pose estimation that operates on unknown object instances within a known category using a single RGB image as input. The proposed network performs 2D object detection, detects 2D key…
NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）
We propose a single-stage, category-level 6-DoF pose estimation algorithm that simultaneously detects and tracks instances of objects within a known category. Our method takes as input the previous and current frame from a monocular RGB video, as well as predictions from the previous frame, to predict the bounding cuboid and 6-DoF pose (up to scale). Internally, a deep network predicts distributions over object keypoints (vertices of the bounding cuboid) in image coordinates, after which a novel probabilistic filtering process integrates across estimates before computing the final pose using P…
Contemporary grasp detection approaches employ deep learning to achieve robustness to sensor and object model uncertainty. The two dominant approaches design either grasp-quality scoring or anchor-based grasp recognition networks. This paper presents a different approach to grasp detection by treating it as keypoint detection in image-space. The deep network detects each grasp candidate as a pair of keypoints, convertible to the grasp representationg = {x, y, w, θ} T , rather than a triplet or quartet of corner points. Decreasing the detection difficulty by grouping keypoints into pairs boosts…
PCKRF: Point Cloud Completion and Keypoint Refinement With Fusion Data for 6D Pose Estimation
Faculty of Information Technology, Beijing University of Technology（北京工业大学信息学部）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Beijing Institute of Technology（北京理工大学）；Shanghai Jiao Tong University（上海交通大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
We present KDFNet, a novel method for 6D object pose estimation from RGB images. To handle occlusion, many recent works have proposed to localize 2D keypoints through pixel-wise voting and solve a Perspective-n-Point (PnP) problem for pose estimation, which achieves leading performance. However, such voting process is direction-based and cannot handle long and thin objects where the direction intersections cannot be robustly found. To address this problem, we propose a novel continuous representation called Keypoint Distance Field (KDF) for projected 2D keypoint locations. Formulated as a 2D a…
Estimating relative camera poses from consecutive frames is a fundamental problem in visual odometry (VO) and simultaneous localization and mapping (SLAM), where classic methods consisting of hand-crafted features and sampling-based outlier rejection have been a dominant choice for over a decade. Although multiple works propose to replace these modules with learning-based counterparts, most have not yet been as accurate, robust and generalizable as conventional methods. In this paper, we design an end-to-end trainable framework consisting of learnable modules for detection, feature extraction,…
Can a robot manipulate intra-category unseen objects in arbitrary poses with the help of a mere demonstration of grasping pose on a single object instance? In this paper, we try to address this intriguing challenge by using USEEK, an unsupervised SE(3)-equivariant keypoints method that enjoys alignment across instances in a category, to perform generalizable manipulation. USEEK follows a teacher-student structure to decouple the unsupervised keypoint discovery and SE(3)-equivariant keypoint detection. With USEEK in hand, the robot can infer the category-level task-relevant object frames in an …
ComPose: A Unified Completion-Pose Framework for Robust Category-Level Object Pose Estimation
University of Science and Technology of China（中国科学技术大学）；National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory（国家空间科学探测重点实验室，深空探测实验室）；Beijing Institute of Control Engineering（北京控制工程研究所）
University of California San Diego（加州大学圣地亚哥分校）；LIRMM（蒙彼利埃计算机科学、机器人学与微电子实验室）；Univ Montpellier（蒙彼利埃大学）；CNRS（法国国家科学研究中心）
State estimation from measured data is crucial for robotic applications as autonomous systems rely on sensors to capture the motion and localize in the 3D world. Among sensors that are designed for measuring a robot's pose, or for soft robots, their shape, vision sensors are favorable because they are information-rich, easy to set up, and cost-effective. With recent advancements in computer vision, deep learning-based methods no longer require markers for identifying feature points on the robot. However, learning-based methods are data-hungry and hence not suitable for soft and prototyping rob…
Calibration of multi-camera systems, i.e. determining the relative poses between the cameras, is a prerequisite for many tasks in computer vision and robotics. Camera calibration is typically achieved using offline methods that use checkerboard calibration targets. These methods, however, often are cumbersome and lengthy, considering that a new calibration is required each time any camera pose changes. In this work, we propose a novel, marker-free online method for the extrinsic calibration of multiple smart edge sensors, relying solely on 2D human keypoint detections that are computed locally…
Robotic Classification of Divers' Swimming States using Visual Pose Keypoints as IMUs
Department of Computer Science & Engineering（计算机科学与工程系）；Minnesota Robotics Institute (MnRI)（明尼苏达州机器人研究所）；University of Minnesota–Twin Cities（明尼苏达大学-双城分校）；Department of Computer Science（计算机科学系）；Carnegie Mellon University（卡内基梅隆大学）
3D Affordance Keypoint Detection for Robotic Manipulation
Advanced Robotics Centre, National University of Singapore（新加坡国立大学先进机器人中心）；Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）
The estimation of 6D poses of rigid objects is a fundamental problem in computer vision. Traditionally pose estimation is concerned with the determination of a single best estimate. However, a single estimate is unable to express visual ambiguity, which in many cases is unavoidable due to object symmetries or occlusion of identifying features. Inability to account for ambiguities in pose can lead to failure in subsequent methods, which is unacceptable when the cost of failure is high. Estimates of full pose distributions are, contrary to single estimates, well suited for expressing uncertainty…
The present project deals with the elastostatic modeling and calibration experiment of spacial industrial manipulators using an optimal selection of measurements pose, for the calibration procedure, the optimal pose selection aims to the efficiency improvement of identification procedure for serial manipulators which reduces noise impact on the parameters identification precision, it is usually used for planar manipulators, our work is mainly to extend the approach for a more complicated manipulator in 3D space using a wise decomposition of the spacial manipulator into a set of serial sub-chai…
University of Delaware（特拉华大学）；Bosch Research North America（博世北美研究院）；Technical University of Munich（慕尼黑工业大学）
We propose a keypoint-based object-level SLAM framework that can provide globally consistent 6DoF pose estimates for symmetric and asymmetric objects alike. To the best of our knowledge, our system is among the first to utilize the camera pose information from SLAM to provide prior knowledge for tracking keypoints on symmetric objects -- ensuring that new measurements are consistent with the current 3D scene. Moreover, our semantic keypoint network is trained to predict the Gaussian covariance for the keypoints that captures the true error of the prediction, and thus is not only useful as a we…
Stanford University（斯坦福大学）；Robotics at Google（谷歌机器人部门）
Estimating the 3D pose of desktop objects is crucial for applications such as robotic manipulation. Many existing approaches to this problem require a depth map of the object for both training and prediction, which restricts them to opaque, lambertian objects that produce good returns in an RGBD sensor. In this paper we forgo using a depth sensor in favor of raw stereo input. We address two problems: first, we establish an easy method for capturing and labeling 3D keypoints on desktop objects with an RGB camera; and second, we develop a deep neural network, called $KeyPose$, that learns to acc…
KTH Royal Institute of Technology（KTH皇家理工学院）；Yale University（耶鲁大学）；Örebro University（厄勒布鲁大学）
We address the problem of motion planning for a robotic manipulator with the task to place a grasped object in a cluttered environment. In this task, we need to locate a collision-free pose for the object that a) facilitates the stable placement of the object, b) is reachable by the robot manipulator and c) optimizes a user-given placement objective. Because of the placement objective, this problem is more challenging than classical motion planning where the target pose is defined from the start. To solve this task, we propose an anytime algorithm that integrates sampling-based motion planning…
This paper presents a kinematic definition of a serialized Stewart platform designed for autonomous in-space assembly called an Assembler. The Assemblers architecture describes problems inherent to the inverse kinematics of over-actuated mixed kinematic systems. This paper also presents a methodology for optimizing poses. In order to accomplish this with the Assembler system, an algorithm for finding a feasible solution to its inverse kinematics was developed with a wrapper for a nonlinear optimization algorithm designed to minimize the magnitude of forces incurred by each actuator. A simulate…
CSAIL, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task -- e.g. knowing the pose and s…
Federal University of Alagoas（阿拉戈斯联邦大学）；Federal University of Minas Gerais（米纳斯吉拉斯联邦大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper presents a solution based on dual quaternion algebra to the general problem of pose (i.e., position and orientation) consensus for systems composed of multiple rigid-bodies. The dual quaternion algebra is used to model the agents' poses and also in the distributed control laws, making the proposed technique easily applicable to time-varying formation control of general robotic systems. The proposed pose consensus protocol has guaranteed convergence when the interaction among the agents is represented by directed graphs with directed spanning trees, which is a more general result whe…
PCHands: PCA-based Hand Pose Synergy Representation on Manipulators with N-DoF
Humanoid Sensing and Perception (HSP), Istituto Italiano di Tecnologia (IIT)（人形感知与感知（HSP），意大利技术研究所（IIT））；Dipartimento di Informatica, Bioingegneria, Robotica e Ingegneria dei Sistemi, University of Genoa（信息学院、生物工程、机器人与系统工程系，热那亚大学）
TU Wien（维也纳工业大学）；University of Liverpool（利物浦大学）；Austrian Institute of Technology GmbH (AIT)（奥地利技术研究院有限公司）
Solving the analytical inverse kinematics (IK) of redundant manipulators in real time is a difficult problem in robotics since its solution for a given target pose is not unique. Moreover, choosing the optimal IK solution with respect to application-specific demands helps to improve the robustness and to increase the success rate when driving the manipulator from its current configuration towards a desired pose. This is necessary, especially in high-dynamic tasks like catching objects in mid-flights. To compute a suitable target configuration in the joint space for a given target pose in the t…
Ecole Centrale de Lille（里尔中央理工学院）；Beihang University（北京航空航天大学）
A high redundant non-holonomic humanoid mobile dual-arm manipulator system is presented in this paper where the motion planning to realize "human-like" autonomous navigation and manipulation tasks is studied. Firstly, an improved MaxiMin NSGA-II algorithm, which optimizes five objective functions to solve the problems of singularity, redundancy, and coupling between mobile base and manipulator simultaneously, is proposed to design the optimal pose to manipulate the target object. Then, in order to link the initial pose and that optimal pose, an off-line motion planning algorithm is designed. I…
SPARK: Low Latency Single-Camera 3D Pose Estimation for Autonomous Racing using Keypoints
Technical University of Munich（慕尼黑工业大学）；School of Engineering & Design, Department of Mobility Systems Engineering, Institute of Automotive Technology（工程与设计学院，移动系统工程系，汽车技术研究所）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）
This paper aims to address the open problem of designing a globally stable vision-based controller for robot manipulators. Accordingly, based on a hybrid mechanism, this paper proposes a novel task-space control law attained by taking the gradient of a potential function in SE(3). The key idea is to employ the Visual Simultaneous Localization and Mapping (VSLAM) algorithm to estimate a robot pose. The estimated robot pose is then used in the proposed hybrid controller as feedback information. Invoking Barbalats lemma and Lyapunov's stability theorem, it is guaranteed that the resulting closed-…
