Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance
Universität Hamburg（汉堡大学）；Agile Robots AG（敏捷机器人公司）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
Currently, task-oriented grasp detection approaches are mostly based on pixel-level affordance detection and semantic segmentation. These pixel-level approaches heavily rely on the accuracy of a 2D affordance mask, and the generated grasp candidates are restricted to a small workspace. To mitigate these limitations, we first construct a novel affordance-based grasp dataset and propose a 6-DoF task-oriented grasp detection framework, which takes the observed object point cloud as input and predicts diverse 6-DoF grasp poses for different tasks. Specifically, our implicit estimation network and visual affordance network in this framework could directly predict coarse grasp candidates, and corresponding 3D affordance heatmap for each potential task, respectively. Furthermore, the grasping scores from coarse grasps are combined with heatmap values to generate more accurate and finer candidates. Our proposed framework shows significant improvements compared to baselines for existing and novel objects on our simulation dataset. Although our framework is trained based on the simulated objects and environment, the final generated grasp candidates can be accurately and stably executed in real robot experiments when the object is randomly placed on a support surface.
Language-Conditioned Affordance-Pose Detection in 3D Point Clouds
FPT Software AI Center（FPT软件人工智能中心）；TU Wien（维也纳工业大学）；Imperial College London（帝国理工学院）；University of Arkansas（阿肯色大学）；Ton Duc Thang University（孙德胜大学）；University of Science, Vietnam（越南科学大学）；University of Liverpool（利物浦大学）
Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance
FPT Software AI Center（FPT软件AI中心）；TU Wien（维也纳技术大学）；AIT GmbH（AIT有限公司）；Imperial College London（伦敦帝国学院）；Physical Intelligence（物理智能公司）；University of Arkansas（阿肯色大学）；Ton Duc Thang University（同塔大学）；University of Liverpool（利物浦大学）
TD-TOG Dataset: Benchmarking Zero-Shot and One-Shot Task-Oriented Grasping for Object Generalization
University of Aberdeen（阿伯丁大学）；Bank of Valletta（瓦莱塔银行）；Universite Claude Bernard Lyon 1（里昂第一大学）；CNRS（法国国家科学研究中心）；École Centrale de Lyon（里昂中央理工学院）；INSA Lyon（里昂国立应用科学学院）；Université Lumière Lyon 2（里昂第二大学）；Technical University of Applied Sciences Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用技术大学）
University of Michigan（密歇根大学）；Google（谷歌）；Google Brain（谷歌大脑）；X Inc（X公司）
This paper focuses on the problem of learning 6-DOF grasping with a parallel jaw gripper in simulation. We propose the notion of a geometry-aware representation in grasping based on the assumption that knowledge of 3D geometry is at the heart of interaction. Our key idea is constraining and regularizing grasping interaction learning through 3D geometry prediction. Specifically, we formulate the learning of deep geometry-aware grasping model in two steps: First, we learn to build mental geometry-aware representation by reconstructing the scene (i.e., 3D occupancy grid) from RGBD input via gener…
ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera
Samsung R&D Institute China-Beijing（三星中国北京研发中心）；CFCS, School of Computer Science, Peking University（计算机科学系，北京大学）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）
University of Edinburgh（爱丁堡大学）；Heriot-Watt University（赫瑞-瓦特大学）
Reasoning about object affordances allows an autonomous agent to perform generalised manipulation tasks among object instances. While current approaches to grasp affordance estimation are effective, they are limited to a single hypothesis. We present an approach for detection and extraction of multiple grasp affordances on an object via visual input. We define semantics as a combination of multiple attributes, which yields benefits in terms of generalisation for grasp affordance prediction. We use Markov Logic Networks to build a knowledge base graph representation to obtain a probability dist…
University of Edinburgh（爱丁堡大学）；Heriot-Watt University（赫瑞-瓦特大学）
Reasoning about object grasp affordances allows an autonomous agent to estimate the most suitable grasp to execute a task. While current approaches for estimating grasp affordances are effective, their prediction is driven by hypotheses on visual features rather than an indicator of a proposal's suitability for an affordance task. Consequently, these works cannot guarantee any level of performance when executing a task and, in fact, not even ensure successful task completion. In this work, we present a pipeline for SAGAT based on prior experiences. We visually detect a grasp affordance region …
Perception-for-grasping is a challenging problem in robotics. Inexpensive range sensors such as the Microsoft Kinect provide sensing capabilities that have given new life to the effort of developing robust and accurate perception methods for robot grasping. This paper proposes a new approach to localizing enveloping grasp affordances in 3-D point clouds efficiently. The approach is based on modeling enveloping grasp affordances as a cylindrical shells that corresponds to the geometry of the robot hand. A fast and accurate fitting method for quadratic surfaces is the core of our approach. An ev…
AffordDexGrasp: Open-set Language-guided Dexterous Grasp with Generalizable-Instructive Affordance
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China（教育部人工智能与高级计算重点实验室）
IEETA - University of Aveiro（阿威罗大学电子与电信研究所）；University of Groningen（格罗宁根大学）
Service robots are expected to autonomously and efficiently work in human-centric environments. For this type of robots, object perception and manipulation are challenging tasks due to need for accurate and real-time response. This paper presents an interactive open-ended learning approach to recognize multiple objects and their grasp affordances concurrently. This is an important contribution in the field of service robots since no matter how extensive the training data used for batch learning, a robot might always be confronted with an unknown object when operating in human-centric environme…
Affordance-Guided Enveloping Grasp Demonstration Toward Non-destructive Disassembly of Pinch-Infeasible Mating Parts
Department of Systems Innovation, Graduate School of Engineering Science, The University of Osaka（大阪大学工学研究院系统创新部门）；Industrial Cyber-physical Systems Research Center, The National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院工业网络物理系统研究中心）
Artificial intelligence is essential to succeed in challenging activities that involve dynamic environments, such as object manipulation tasks in indoor scenes. Most of the state-of-the-art literature explores robotic grasping methods by focusing exclusively on attributes of the target object. When it comes to human perceptual learning approaches, these physical qualities are not only inferred from the object, but also from the characteristics of the surroundings. This work proposes a method that includes environmental context to reason on an object affordance to then deduce its grasping regio…
Self-supervised grasp learning, i.e., learning to grasp by trial and error, has made great progress. However, it is still time-consuming to train such a model and also a challenge to apply it in practice. This work presents an accelerating method of robotic grasp learning via pretraining with coarse affordance maps of objects to be grasped based on a quite small dataset. A model generated through pre-training is harnessed as an initialization policy to warmly start grasp learning so as to guide a robot to capture more effective rewards at the beginning of training. An object in its coarse affo…
Institute of Knowledge Based Engineering, Shanghai Jiao Tong University（上海交通大学知识工程研究所）
In this paper we study grasp problem in dense cluster, a challenging task in warehouse logistics scenario. By introducing a two-step robust suction affordance detection method, we focus on using vacuum suction pad to clear up a box filled with seen and unseen objects. Two CNN based neural networks are proposed. A Fast Region Estimation Network (FRE-Net) predicts which region contains pickable objects, and a Suction Grasp Point Affordance network (SGPA-Net) determines which point in that region is pickable. So as to enable such two networks, we design a self-supervised learning pipeline to accu…
AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Peking University（北京大学）
MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations
The Department of Electronic Engineering, Tsinghua University, Beijing 100084, China（清华大学电子工程系，北京100084，中国）；The Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China（香港科学与技术大学电子与计算机工程系，香港，中国）
University of California, Berkeley（加州大学伯克利分校）；FANUC America Corporation（发那科美国公司）
Proposing grasp poses for novel objects is an essential component for any robot manipulation task. Planning six degrees of freedom (DoF) grasps with a single camera, however, is challenging due to the complex object shape, incomplete object information, and sensor noise. In this paper, we present a 6-DoF contrastive grasp proposal network (CGPN) to infer 6-DoF grasps from a single-view depth image. First, an image encoder is used to extract the feature map from the input depth image, after which 3-DoF grasp regions are proposed from the feature map with a rotated region proposal network. Featu…
DORA: Object Affordance-Guided Reinforcement Learning for Dexterous Robotic Manipulation
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, Universität Hamburg（技术多模态系统（TAMS），信息学院，汉堡大学）；Agile Robots AG（敏捷机器人公司）；Technical University of Munich（慕尼黑技术大学）
A key challenge in robot teaching is grasp-type recognition with a single RGB image and a target object name. Here, we propose a simple yet effective pipeline to enhance learning-based recognition by leveraging a prior distribution of grasp types for each object. In the pipeline, a convolutional neural network (CNN) recognizes the grasp type from an RGB image. The recognition result is further corrected using the prior distribution (i.e., affordance), which is associated with the target object name. Experimental results showed that the proposed method outperforms both a CNN-only and an afforda…
National University of Defense Technology（国防科技大学）；University of Maryland at College Park（马里兰大学帕克分校）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
We present a learning-based method for representing grasp poses of a high-DOF hand using neural networks. Due to redundancy in such high-DOF grippers, there exists a large number of equally effective grasp poses for a given target object, making it difficult for the neural network to find consistent grasp poses. We resolve this ambiguity by generating an augmented dataset that covers many possible grasps for each target object and train our neural networks using a consistency loss function to identify a one-to-one mapping from objects to grasp poses. We further enhance the quality of neural-ne…
Sun Yat-sen University（中山大学）；SenseTime Group Limited（商汤集团有限公司）
Data-driven approach for grasping shows significant advance recently. But these approaches usually require much training data. To increase the efficiency of grasping data collection, this paper presents a novel grasp training system including the whole pipeline from data collection to model inference. The system can collect effective grasp sample with a corrective strategy assisted by antipodal grasp rule, and we design an affordance interpreter network to predict pixelwise grasp affordance map. We define graspability, ungraspability and background as grasp affordances. The key advantage of ou…
School of CS & AI, Arizona State University（亚利桑那州立大学计算机科学与人工智能学院）
Conventional works that learn grasping affordance from demonstrations need to explicitly predict grasping configurations, such as gripper approaching angles or grasping preshapes. Classic motion planners could then sample trajectories by using such predicted configurations. In this work, our goal is instead to fill the gap between affordance discovery and affordance-based policy learning by integrating the two objectives in an end-to-end imitation learning framework based on deep neural networks. From a psychological perspective, there is a close association between attention and affordance. T…
General robot grasping in clutter requires the ability to synthesize grasps that work for previously unseen objects and that are also robust to physical interactions, such as collisions with other objects in the scene. In this work, we design and train a network that predicts 6 DOF grasps from 3D scene information gathered from an on-board sensor such as a wrist-mounted depth camera. Our proposed Volumetric Grasping Network (VGN) accepts a Truncated Signed Distance Function (TSDF) representation of the scene and directly outputs the predicted grasp quality and the associated gripper orientatio…
Triplane Grasping: Efficient 6-DoF Grasping with Single RGB Images
Computer Science Department, Faculty of Science and Engineering, Swansea University（计算机科学系，科学与工程学院，斯旺西大学）；Department of Computer Science, Durham University（计算机科学系，达勒姆大学）
6-DoF Grasp Detection in Clutter with Enhanced Receptive Field and Graspable Balance Sampling
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications（北京邮电大学智能工程与自动化学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
