This paper looks into the problem of grasping region localization along with suitable pose from a cluttered environment without any a priori knowledge of the object geometry. This end-to-end method detects the handles from a single frame of input sensor. The pipeline starts with the creation of multiple surface segments to detect the required gap in the first stage, and eventually helps in detecting boundary lines. Our novelty lies in the fact that we have merged color based edge and depth edge in order to get more reliable boundary points through which a pair of boundary line is fitted. Also this information is used to validate the handle by measuring the angle between the boundary lines and also by checking for amy potential occlusion. In addition, we also proposed an optimizing cost function based method to choose the best handle from a set of valid handles. The method proposed is tested on real-life datasets and is found to out form state of the art methods in terms of precision.
ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes
CFCS, School of Computer Science, Peking University（计算机学院，北京大学）；PKU-AgiBot Lab（北京大学阿吉机器人实验室）；PKU-PsiBot Lab（北京大学Psi机器人实验室）；Princeton University（普林斯顿大学）
GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes
Korea Institute of Machinery & Materials (KIMM)（韩国机械材料研究院）；Gwangju Institute of Science and Technology (GIST)（全州科学技术院）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
The food packaging industry handles an immense variety of food products with wide-ranging shapes and sizes, even within one kind of food. Menus are also diverse and change frequently, making automation of pick-and-place difficult. A popular approach to bin-picking is to first identify each piece of food in the tray by using an instance segmentation method. However, human annotations to train these methods are unreliable and error-prone since foods are packed close together with unclear boundaries and visual similarity making separation of pieces difficult. To address this problem, we propose a…
Suction is an important solution for the longstanding robotic grasping problem. Compared with other kinds of grasping, suction grasping is easier to represent and often more reliable in practice. Though preferred in many scenarios, it is not fully investigated and lacks sufficient training data and evaluation benchmarks. To address that, firstly, we propose a new physical model to analytically evaluate seal formation and wrench resistance of a suction grasping, which are two key aspects of grasp success. Secondly, a two-step methodology is adopted to generate annotations on a large-scale datas…
East China Normal University（华东师范大学）；Midea Group（美的集团）；Guangdong Polytechnic Normal University（广东技术师范大学）
In this paper, we propose a novel representation for grasping using contacts between multi-finger robotic hands and objects to be manipulated. This representation significantly reduces the prediction dimensions and accelerates the learning process. We present an effective end-to-end network, CMG-Net, for grasping unknown objects in a cluttered environment by efficiently predicting multi-finger grasp poses and hand configurations from a single-shot point cloud. Moreover, we create a synthetic grasp dataset that consists of five thousand cluttered scenes, 80 object categories, and 20 million ann…
Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）；NVIDIA（英伟达）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
6D grasping in cluttered scenes is a longstanding problem in robotic manipulation. Open-loop manipulation pipelines may fail due to inaccurate state estimation, while most end-to-end grasping methods have not yet scaled to complex scenes with obstacles. In this work, we propose a new method for end-to-end learning of 6D grasping in cluttered scenes. Our hierarchical framework learns collision-free target-driven grasping based on partial point cloud observations. We learn an embedding space to encode expert grasping plans during training and a variational autoencoder to sample diverse grasping …
Although, in the task of grasping via a data-driven method, closed-loop feedback and predicting 6 degrees of freedom (DoF) grasp rather than conventionally used 4DoF top-down grasp are demonstrated to improve performance individually, few systems have both. Moreover, the sequential property of that task is hardly dealt with, while the approaching motion necessarily generates a series of observations. Therefore, this paper synthesizes three approaches and suggests a closed-loop framework that can predict the 6DoF grasp in a heavily cluttered environment from continuously received vision observa…
Wuhan University（武汉大学）；University of the West of England（西英格兰大学）
The ability to segment unknown objects in cluttered scenes has a profound impact on robot grasping. The rise of deep learning has greatly transformed the pipeline of robotic grasping from model-based approach to data-driven stream, which generally requires a large scale of grasping data either collected in simulation or from real-world examples. In this paper, we proposed a sim-to-real framework to transfer the object segmentation model learned in simulation to the real-world. First, data samples are collected in simulation, including RGB, 6D pose, and point cloud. Second, we also present a GA…
Wood logs picking is a challenging task to automate. Indeed, logs usually come in cluttered configurations, randomly orientated and overlapping. Recent work on log picking automation usually assume that the logs' pose is known, with little consideration given to the actual perception problem. In this paper, we squarely address the latter, using a data-driven approach. First, we introduce a novel dataset, named TimberSeg 1.0, that is densely annotated, i.e., that includes both bounding boxes and pixel-level mask annotations for logs. This dataset comprises 220 images with 2500 individually segm…
Grasp detection in cluttered scenes is a very challenging task for robots. Generating synthetic grasping data is a popular way to train and test grasp methods, as is Dex-net and GraspNet; yet, these methods generate training grasps on 3D synthetic object models, but evaluate at images or point clouds with different distributions, which reduces performance on real scenes due to sparse grasp labels and covariate shift. To solve existing problems, we propose a novel on-policy grasp detection method, which can train and test on the same distribution with dense pixel-level grasp labels generated on…
GraspView: Active Perception Scoring and Best-View Optimization for Robotic Grasping in Cluttered Environments
Peng Cheng Laboratory（鹏城实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院）；Shenzhen Graduate School, Peking University（深圳研究生院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；X-Era AI Lab（X-Era AI实验室）
CADGrasp: Learning Contact and Collision Aware General Dexterous Grasping in Cluttered Scenes
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（前沿计算研究中心，计算机学院，北京大学）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家级重点实验室，计算机学院，北京大学）
University of Groningen（格罗宁根大学）；University of Edinburgh（爱丁堡大学）
Nowadays robots play an increasingly important role in our daily life. In human-centered environments, robots often encounter piles of objects, packed items, or isolated objects. Therefore, a robot must be able to grasp and manipulate different objects in various situations to help humans with daily tasks. In this paper, we propose a multi-view deep learning approach to handle robust object grasping in human-centric domains. In particular, our approach takes a point cloud of an arbitrary object as an input, and then, generates orthographic views of the given object. The obtained views are fina…
ContactDexNet: Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-object Contact Semantic Mapping
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, Universität Hamburg（乌尔姆大学信息学院技术多模态系统部门）；Agile Robots AG（敏捷机器人公司）；Technical University of Munich（慕尼黑技术大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Object grasping in cluttered scenes is a widely investigated field of robot manipulation. Most of the current works focus on estimating grasp pose from point clouds based on an efficient single-shot grasp detection network. However, due to the lack of geometry awareness of the local grasping area, it may cause severe collisions and unstable grasp configurations. In this paper, we propose a two-stage grasp pose refinement network which detects grasps globally while fine-tuning low-quality grasps and filtering noisy grasps locally. Furthermore, we extend the 6-DoF grasp with an extra dimension a…
Tsinghua University（清华大学）；Shenzhen Academy of Robotics（深圳机器人研究院）
In this paper, a novel robotic grasping system is established to automatically pick up objects in cluttered scenes. A composite robotic hand composed of a suction cup and a gripper is designed for grasping the object stably. The suction cup is used for lifting the object from the clutter first and the gripper for grasping the object accordingly. We utilize the affordance map to provide pixel-wise lifting point candidates for the suction cup. To obtain a good affordance map, the active exploration mechanism is introduced to the system. An effective metric is designed to calculate the reward for…
D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping
Physical Superintelligence (PSI) Lab, University of Southern California（南加州大学物理超智能实验室）；Viterbi School of Engineering, University of Southern California（南加州大学韦伯尔工程学院）；Department of EECS, University of California, Berkeley（加州大学伯克利分校电子工程与计算机科学系）
Clutt3R-Seg: Sparse-view 3D Instance Segmentation for Language-grounded Grasping in Cluttered Scenes
Dept. of Mechanical Engineering, SNU（机械工程系，首尔国立大学）；Interdisciplinary Program in Artificial Intelligence, SNU（人工智能跨学科项目，首尔国立大学）；Robotics Lab, Hyundai Motor Company（Hyundai Motor Company机器人实验室）
ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera
Samsung R&D Institute China-Beijing（三星中国北京研发中心）；CFCS, School of Computer Science, Peking University（计算机科学系，北京大学）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）
Region-aware Grasp Framework with Normalized Grasp Space for Efficient 6-DoF Grasping
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Shanghai AI Laboratory（上海人工智能实验室）
Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所（IPA））
In this paper, we introduce a novel learning-based approach for grasping known rigid objects in highly cluttered scenes and precisely placing them based on depth images. Our Placement Quality Network (PQ-Net) estimates the object pose and the quality for each automatically generated grasp pose for multiple objects simultaneously at 92 fps in a single forward pass of a neural network. All grasping and placement trials are executed in a physics simulation and the gained experience is transferred to the real world using domain randomization. We demonstrate that our policy successfully transfers t…
