TransSC: Transformer-based Shape Completion for Grasp Evaluation
Universität Hamburg（汉堡大学）；Agile Robots AG（敏捷机器人公司）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
Currently, robotic grasping methods based on sparse partial point clouds have attained a great grasping performance on various objects while they often generate wrong grasping candidates due to the lack of geometric information on the object. In this work, we propose a novel and robust shape completion model (TransSC). This model has a transformer-based encoder to explore more point-wise features and a manifold-based decoder to exploit more object details using a partial point cloud as input.
Quantitative experiments verify the effectiveness of the proposed shape completion network and demonstrate it outperforms existing methods. Besides, TransSC is integrated into a grasp evaluation network to generate a set of grasp candidates. The simulation experiment shows that TransSC improves the grasping generation result compared to the existing shape completion baselines. Furthermore, our robotic experiment shows that with TransSC the robot is more successful in grasping objects that are randomly placed on a support surface.
This work provides an architecture to enable robotic grasp planning via shape completion. Shape completion is accomplished through the use of a 3D convolutional neural network (CNN). The network is trained on our own new open source dataset of over 440,000 3D exemplars captured from varying viewpoints. At runtime, a 2.5D pointcloud captured from a single point of view is fed into the CNN, which fills in the occluded regions of the scene, allowing grasps to be planned and executed on the completed object. Runtime shape completion is very rapid because most of the computational costs of shape co…
We present a method for planning robust grasps over uncertain shape completed objects. For shape completion, a deep neural network is trained to take a partial view of the object as input and outputs the completed shape as a voxel grid. The key part of the network is dropout layers which are enabled not only during training but also at run-time to generate a set of shape samples representing the shape uncertainty through Monte Carlo sampling. Given the set of shape completed objects, we generate grasp candidates on the mean object shape but evaluate them based on their joint performance in ter…
Current end-to-end grasp planning methods propose grasps in the order of seconds that attain high grasp success rates on a diverse set of objects, but often by constraining the workspace to top-grasps. In this work, we present a method that allows end-to-end top-grasp planning methods to generate full six-degree-of-freedom grasps using a single RGB-D view as input. This is achieved by estimating the complete shape of the object to be grasped, then simulating different viewpoints of the object, passing the simulated viewpoints to an end-to-end grasp generation method, and finally executing the …
PCF-Grasp: Converting Point Completion to Geometry Feature to Enhance 6-DoF Grasp
State Key Laboratory of Robotics and System at Harbin Institute of Technology（哈尔滨工业大学机器人系统国家重点实验室）；Lanzhou University of Technology（兰州理工大学）；Department of Computer Science（计算机科学系）；University of Liverpool（利物浦大学）
Vislab, Institute for Systems and Robotics—Lisboa, Instituto Superior Técnico, Universidade de Lisboa, Portugal（葡萄牙里斯本系统与机器人研究所Vislab，理工学院里斯本高等技术大学，里斯本大学）；Department of Marine, Electrical, Electronic and Telecommunications Engineering, University of Genoa, Italy（意大利热那亚大学海洋、电气、电子与电信工程系）；Pattern Analysis & Computer Vision (PAVIS), Istituto Italiano di Tecnologia (IIT), Genoa, Italy（意大利热那亚理工学院模式分析与计算机视觉（PAVIS））；Deep Visual Learning (DVL), Fondazione Bruno Kessler, Trento, Italy（意大利特伦托布鲁诺·凯瑟基金会深度视觉学习（DVL））
Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；AI Institute（人工智能研究院）
Transparent objects are common in our daily life and frequently handled in the automated production line. Robust vision-based robotic grasping and manipulation for these objects would be beneficial for automation. However, the majority of current grasping algorithms would fail in this case since they heavily rely on the depth image, while ordinary depth sensors usually fail to produce accurate depth information for transparent objects owing to the reflection and refraction of light. In this work, we address this issue by contributing a large-scale real-world dataset for transparent object dept…
PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation
School of Automation, Guangdong University of Technology（广东工业大学自动化学院）；University of Liverpool（利物浦大学）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算学系）
Reliable robotic grasping in unstructured environments is a crucial but challenging task. The main problem is to generate the optimal grasp of novel objects from partial noisy observations. This paper presents an end-to-end grasp detection network taking one single-view point cloud as input to tackle the problem. Our network includes three stages: Score Network (SN), Grasp Region Network (GRN), and Refine Network (RN). Specifically, SN regresses point grasp confidence and selects positive points with high confidence. Then GRN conducts grasp proposal prediction on the selected positive points. …
Grasping for novel objects is important for robot manipulation in unstructured environments. Most of current works require a grasp sampling process to obtain grasp candidates, combined with local feature extractor using deep learning. This pipeline is time-costly, expecially when grasp points are sparse such as at the edge of a bowl. In this paper, we propose an end-to-end approach to directly predict the poses, categories and scores (qualities) of all the grasps. It takes the whole sparse point clouds as the input and requires no sampling or search process. Moreover, to generate training data…
ROG-Grasp: Root-Oriented Geometry for Robotic Grasping and Placement
Department of Electrical and Computer Engineering, Drexel University（德雷塞尔大学电气与计算机工程系）；Virginia Seafood Agricultural Research and Extension Center, and Department of Biological Systems Engineering, Virginia Tech（弗吉尼亚理工学院生物系统工程系和弗吉尼亚海鲜农业研究与推广中心）；Amazon Store Foundation AI (SFAI)（亚马逊商店基金会人工智能（SFAI））
LangGrasp: Leveraging Fine-Tuned LLMs for Language Interactive Robot Grasping with Ambiguous Instructions
School of Computer Science and Technology（计算机科学与技术学院）；Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System（湖北省智能信息处理与实时工业系统重点实验室）；Institute of Robotics and Intelligent Systems（机器人与智能系统研究院）；Wuhan University of Science and Technology（武汉理工大学）
Language-Guided Grasping under Partial Observation for Mobile Manipulation in Field Inspection and Maintenance
Department of Mechanical Engineering, Federal University of Uberlândia（联邦大学伯南迪利亚机械工程系）；Department of Mechanical Engineering, University of São Paulo（圣保罗大学机械工程系）；Graduate School of Information Sciences, Tohoku University（东北大学信息科学研究生院）；Faculdade Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein（艾伯特·爱因斯坦以色列教学与研究学院，艾伯特·爱因斯坦医院）
Many robotic tasks involving some form of 3D visual perception greatly benefit from a complete knowledge of the working environment. However, robots often have to tackle unstructured environments and their onboard visual sensors can only provide incomplete information due to limited workspaces, clutter or object self-occlusion. In recent years, deep learning architectures for shape completion have begun taking traction as effective means of inferring a complete 3D object representation from partial visual data. Nevertheless, most of the existing state-of-the-art approaches provide a fixed outp…
Grasp, Slide, Roll: Comparative Analysis of Contact Modes for Tactile-Based Shape Reconstruction
Amazon Fulfillment Technologies & Robotics（亚马逊履行技术与机器人）；Robotics Institute at Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院）
ARC Centre for Excellence for Robot Vision（澳大利亚研究理事会机器人视觉卓越中心）；Monash University（莫纳什大学）
Grasping is the dominant approach for robot manipulation, but only a single object can be grasped at a time. Nonprehensile manipulation offers richer set of interactions, however state-of-the-art is limited to using the end-effector only. We propose using a robot link (forearm) to push multiple objects at once. In a simulated task where the robot's task is to sort two kinds of objects into their respective goal regions, we show that a greedy strategy that uses a combination of forearm pushes and pick and place operations reduces task completion time by %28 compared to picking and placing each …
Measuring Uncertainty in Shape Completion to Improve Grasp Quality
Vislab, ISR-Lisboa, Instituto Superior Técnico, Universidade de Lisboa, Portugal（Vislab、ISR-里斯本、里斯本技术高等学院、里斯本大学、葡萄牙）；PAVIS, Fondazione Istituto Italiano di Tecnologia, Genoa, Italy（PAVIS、意大利技术研究院基金会、热那亚、意大利）；Leonardo, Genoa, Italy（Leonardo、热那亚、意大利）
Both goal-agnostic and goal-oriented tasks have practical value for robotic grasping: goal-agnostic tasks target all objects in the workspace, while goal-oriented tasks aim at grasping pre-assigned goal objects. However, most current grasping methods are only better at coping with one task. In this work, we propose a bifunctional push-grasping synergistic strategy for goal-agnostic and goal-oriented grasping tasks. Our method integrates pushing along with grasping to pick up all objects or pre-assigned goal objects with high action efficiency depending on the task requirement. We introduce a b…
HGACNet: Hierarchical Graph Attention Network for Cross-Modal Point Cloud Completion
Robotics Research Centre of the School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（南洋理工大学机械与航空航天工程学院机器人研究中心）；College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China（西安建筑科技大学信息与控制工程学院）
SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping
Beijing University of Posts and Telecommunications（北京邮电大学）；Center on Frontiers of Computing Studies, Peking University（北京大学前沿计算研究中心）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Recovering full 3D shapes from partial observations is a challenging task that has been extensively addressed in the computer vision community. Many deep learning methods tackle this problem by training 3D shape generation networks to learn a prior over the full 3D shapes. In this training regime, the methods expect the inputs to be in a fixed canonical form, without which they fail to learn a valid prior over the 3D shapes. We propose SCARP, a model that performs Shape Completion in ARbitrary Poses. Given a partial pointcloud of an object, SCARP learns a disentangled feature representation of…
Khoury College of Computer Sciences, Northeastern University（东北大学科里计算机科学学院）
We consider robotic pick-and-place of partially visible, novel objects, where goal placements are non-trivial, e.g., tightly packed into a bin. One approach is (a) use object instance segmentation and shape completion to model the objects and (b) use a regrasp planner to decide grasps and places displacing the models to their goals. However, it is critical for the planner to account for uncertainty in the perceived models, as object geometries in unobserved areas are just guesses. We account for perceptual uncertainty by incorporating it into the regrasp planner's cost function. We compare sev…
In this paper, we present a method to manipulate unknown objects in-hand using tactile sensing without relying on a known object model. In many cases, vision-only approaches may not be feasible; for example, due to occlusion in cluttered spaces. We address this limitation by introducing a method to reorient unknown objects using tactile sensing. It incrementally builds a probabilistic estimate of the object shape and pose during task-driven manipulation. Our approach uses Bayesian optimization to balance exploration of the global object shape with efficient task completion. To demonstrate the …
GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objects
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization（工业智能与系统优化国家前沿科学中心）
Columbia University（哥伦比亚大学）；U.S. Army Research Laboratory（美国陆军研究实验室）；Robotics at Google（谷歌机器人部门）
While both navigation and manipulation are challenging topics in isolation, many tasks require the ability to both navigate and manipulate in concert. To this end, we propose a mobile manipulation system that leverages novel navigation and shape completion methods to manipulate an object with a mobile robot. Our system utilizes uncertainty in the initial estimation of a manipulation target to calculate a predicted next-best-view. Without the need of localization, the robot then uses the predicted panoramic view at the next-best-view location to navigate to the desired location, capture a secon…
Robots in the real world frequently come across identical objects in dense clutter. When evaluating grasp poses in these scenarios, a target-driven grasping system requires knowledge of spatial relations between scene objects (e.g., proximity, adjacency, and occlusions). To efficiently complete this task, we propose a target-driven grasping system that simultaneously considers object relations and predicts 6-DoF grasp poses. A densely cluttered scene is first formulated as a grasp graph with nodes representing object geometries in the grasp coordinate frame and edges indicating spatial relatio…
Suction Leap-Hand: Suction Cups on a Multi-fingered Hand Enable Embodied Dexterity and In-Hand Teleoperation
School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；School of Physics, Engineering and Technology, University of York（物理、工程与技术学院，约克大学）；Tsinghua University（清华大学）
AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior
The Hong Kong University of Science & Technology（香港科技大学）；The University of Hong Kong（香港大学）；Southern University of Science & Technology（南方科技大学）；Zhejiang University（浙江大学）；LimX Dynamics
University of Edinburgh（爱丁堡大学）；Heriot-Watt University（赫瑞-瓦特大学）
Reasoning about object grasp affordances allows an autonomous agent to estimate the most suitable grasp to execute a task. While current approaches for estimating grasp affordances are effective, their prediction is driven by hypotheses on visual features rather than an indicator of a proposal's suitability for an affordance task. Consequently, these works cannot guarantee any level of performance when executing a task and, in fact, not even ensure successful task completion. In this work, we present a pipeline for SAGAT based on prior experiences. We visually detect a grasp affordance region …
CTU in Prague（布拉格捷克理工大学）；Aalto University（阿尔托大学）；Czech Institute of Informatics, Robotics, and Cybernetics, CTU in Prague（布拉格捷克理工大学捷克信息学、机器人与控制论研究所）
Recent advancements in object shape completion have enabled impressive object reconstructions using only visual input. However, due to self-occlusion, the reconstructions have high uncertainty in the occluded object parts, which negatively impacts the performance of downstream robotic tasks such as grasping. In this work, we propose an active visuo-haptic shape completion method called Act-VH that actively computes where to touch the objects based on the reconstruction uncertainty. Act-VH reconstructs objects from point clouds and calculates the reconstruction uncertainty using IGR, a recent s…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）
Grasping with anthropomorphic robotic hands involves much more hand-object interactions compared to parallel-jaw grippers. Modeling hand-object interactions is essential to the study of multi-finger hand dextrous manipulation. This work presents DVGG, an efficient grasp generation network that takes single-view observation as input and predicts high-quality grasp configurations for unknown objects. In general, our generative model consists of three components: 1) Point cloud completion for the target object based on the partial observation; 2) Diverse sets of grasps generation given the comple…
We focus on the task of goal-oriented grasping, in which a robot is supposed to grasp a pre-assigned goal object in clutter and needs some pre-grasp actions such as pushes to enable stable grasps. However, in this task, the robot gets positive rewards from environment only when successfully grasping the goal object. Besides, joint pushing and grasping elongates the action sequence, compounding the problem of reward delay. Thus, sample inefficiency remains a main challenge in this task. In this paper, a goal-conditioned hierarchical reinforcement learning formulation with high sample efficiency…
Generalized Anthropomorphic Functional Grasping with Minimal Demonstrations
School of Artificial Intelligence, Chinese Academy of Sciences（中国科学院人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）
Trans2Occ: Voxel Occupancy Estimation and Grasp for Transparent Objects from Simulation to Reality
Shanghai AI Laboratory（上海人工智能实验室）；SUSTech（南方科技大学）；CUHK（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Oxford（牛津大学）；Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）
RL-GSBridge: 3D Gaussian Splatting Based Real2Sim2Real Method for Robotic Manipulation Learning
Shenzhen Research Institute of Shanghai Jiao Tong University（上海交通大学深圳研究院）；Shanghai Jiao Tong University（上海交通大学）；School of Information and Control Engineering, China University of Mining and Technology（中国矿业大学信息与控制工程学院）；University of Cambridge（剑桥大学）
LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models
School of Information and Communication Engineering（信息与通信工程学院）；Dalian University of Technology（大连理工大学）；Beta Infinity（贝塔无限）；Nanyang Technological University（南洋理工大学）；Beijing Jiaotong University（北京交通大学）
Efficient Heatmap-Guided 6-Dof Grasp Detection in Cluttered Scenes
Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）
