Deep learning is an established framework for learning hierarchical data representations. While compute power is in abundance, one of the main challenges in applying this framework to robotic grasping has been obtaining the amount of data needed to learn these representations, and structuring the data to the task at hand. Among contemporary approaches in the literature, we highlight key properties that have encouraged the use of deep learning techniques, and in this paper, detail our experience in developing a simulator for collecting cylindrical precision grasps of a multi-fingered dexterous robotic hand.
Grasping deformable objects is not well researched due to the complexity in modelling and simulating the dynamic behavior of such objects. However, with the rapid development of physics-based simulators that support soft bodies, the research gap between rigid and deformable objects is getting smaller. To leverage the capability of such simulators and to challenge the assumption that has guided robotic grasping research so far, i.e., object rigidity, we proposed a deep-learning based approach that generates stiffness-dependent grasps. Our network is trained on purely synthetic data generated fr…
MISCGrasp: Leveraging Multiple Integrated Scales and Contrastive Learning for Enhanced Volumetric Grasping
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Qiyuan Lab（启元实验室）
Siemens（西门子）；University of California, Berkeley（加州大学伯克利分校）；Ambidextrous Laboratories, Inc（Ambidextrous Laboratories公司）
Universal grasping of a diverse range of previously unseen objects from heaps is a grand challenge in e-commerce order fulfillment, manufacturing, and home service robotics. Recently, deep learning based grasping approaches have demonstrated results that make them increasingly interesting for industrial deployments. This paper explores the problem from an automation systems point-of-view. We develop a robotics grasping system using Dex-Net, which is fully integrated at the controller level. Two neural networks are deployed on a novel industrial AI hardware acceleration module close to a PLC wi…
VILAS: A VLA-Integrated Low-cost Architecture with Soft Grasping for Robotic Manipulation
Drexel University（德雷塞尔大学）；Virginia Seafood Agricultural Research and Extension Center（弗吉尼亚海鲜农业研究与推广中心）；Amazon Store Foundation AI (SFAI)（亚马逊商店基金会人工智能（SFAI））
A task-sequencing simulator in robotics manipulation to integrate simulation-for-learning and simulation-for-execution is introduced. Unlike existing machine-learning simulation where a non-decomposed simulation is used to simulate a training scenario, the task-sequencing simulator runs a composed simulation using building blocks. This way, the simulation-for-learning is structured similarly to a multi-step simulation-for-execution. To compose both learning and execution scenarios, a unified trainable-and-composable description of blocks called a concept model is proposed and used. Using the s…
6D robotic grasping beyond top-down bin-picking scenarios is a challenging task. Previous solutions based on 6D grasp synthesis with robot motion planning usually operate in an open-loop setting, which are sensitive to grasp synthesis errors. In this work, we propose a new method for learning closed-loop control policies for 6D grasping. Our policy takes a segmented point cloud of an object from an egocentric camera as input, and outputs continuous 6D control actions of the robot gripper for grasping the object. We combine imitation learning and reinforcement learning and introduce a goal-auxi…
Sequential Multi-Object Grasping with One Dexterous Hand
Thomas Lord Department of Computer Science, University of Southern California, USA（美国南加州大学计算机科学系）；Fudan University, China（复旦大学）；INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria（保加利亚索菲亚大学“圣克莱孟·奥赫里茨基”学院）
Universität Hamburg（汉堡大学）；Agile Robots AG（敏捷机器人公司）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
Currently, robotic grasping methods based on sparse partial point clouds have attained a great grasping performance on various objects while they often generate wrong grasping candidates due to the lack of geometric information on the object. In this work, we propose a novel and robust shape completion model (TransSC). This model has a transformer-based encoder to explore more point-wise features and a manifold-based decoder to exploit more object details using a partial point cloud as input. Quantitative experiments verify the effectiveness of the proposed shape completion network and demonst…
Towards Autonomous Wood-Log Grasping with a Forestry Crane: Simulator and Benchmarking
Automation & Control Institute (ACIN), TU Wien（自动化与控制研究所（ACIN），维也纳技术大学）；AIT Austrian Institute of Technology GmbH（奥地利技术研究院）；Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）
Whole-Body Proprioceptive Morphing: A Modular Soft Gripper for Robust Cross-Scale Grasping
Mechanical Engineering Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机械工程系）；Robotics Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机器人学系）
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Grasping is an essential capability for most robots in practical applications. Soft robotic grippers are considered as a critical part of robotic grasping and have attracted considerable attention in terms of the advantages of the high compliance and robustness to variance in object geometry; however, they are still limited by the corresponding sensing capabilities and actuation mechanisms. We propose a novel soft gripper that looks like a granary with a compliant snap-through bistable mechanism fabricated by integrated mold technology, achieving sensing and actuation purely mechanically. In p…
Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Tsinghua University（清华大学）
Robotic dexterous grasping is the first step to enable human-like dexterous object manipulation and thus a crucial robotic technology. However, dexterous grasping is much more under-explored than object grasping with parallel grippers, partially due to the lack of a large-scale dataset. In this work, we present a large-scale robotic dexterous grasp dataset, DexGraspNet, generated by our proposed highly efficient synthesis method that can be generally applied to any dexterous hand. Our method leverages a deeply accelerated differentiable force closure estimator and thus can efficiently and robu…
Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；KEENON Robotics Co., Ltd.（科语机器人有限公司）；Suzhou Silicon Era Intelligent Technology Co., Ltd.（苏州硅时代智能科技有限公司）；College of Materials, Xiamen University（厦门大学材料学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；ByteDance（字节跳动）；State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院通用人工智能国家重点实验室）；Hubei Automation Institute（湖北省自动化研究所）
Visual-tactile manipulation to collect household waste in outdoor
Instituto Universitario de Investigación Informática, Universidad de Alicante（阿利坎特大学大学信息学研究学院）；Universidad de Alicante（阿利坎特大学）；Universidad Miguel Hernández（米格尔·埃尔南德斯大学）
Hook-Based Aerial Payload Grasping from a Moving Platform
Systems and Control Lab, HUN-REN Institute for Computer Science and Control（系统与控制实验室，HUN-REN计算机科学与控制研究所）；National Laboratory for Autonomous Systems（自主系统国家实验室）；Control Systems Group of the Eindhoven University of Technology（埃因霍温理工大学控制系统组）
School of Information Technologies, University of Sydney（悉尼大学信息技术学院）；ETH Zürich（苏黎世联邦理工学院）
With humankind facing new and increasingly large-scale challenges in the medical and domestic spheres, automation of the service sector carries a tremendous potential for improved efficiency, quality, and safety of operations. Mobile robotics can offer solutions with a high degree of mobility and dexterity, however these complex systems require a multitude of heterogeneous components to be carefully integrated into one consistent framework. This work presents a mobile manipulation system that combines perception, localization, navigation, motion planning and grasping skills into one common wor…
Grasping in cluttered scenes is challenging for robot vision systems, as detection accuracy can be hindered by partial occlusion of objects. We adopt a reinforcement learning (RL) framework and 3D vision architectures to search for feasible viewpoints for grasping by the use of hand-mounted RGB-D cameras. To overcome the disadvantages of photo-realistic environment simulation, we propose a large-scale dataset called Real Embodied Dataset (RED), which includes full-viewpoint real samples on the upper hemisphere with amodal annotation and enables a simulator that has real visual feedback. Based …
NVIDIA（英伟达）；University of Southern California（南加州大学）
We address the problem of directional semantic grasping, that is, grasping a specific object from a specific direction. We approach the problem using deep reinforcement learning via a double deep Q-network (DDQN) that learns to map downsampled RGB input images from a wrist-mounted camera to Q-values, which are then translated into Cartesian robot control commands via the cross-entropy method (CEM). The network is learned entirely on simulated data generated by a custom robot simulator that models both physical reality (contacts) and perceptual quality (high-quality rendering). The reality gap …
A novel integrated method of detection-grasping for specific object based on the box coordinate matching
National Research Base of Intelligent Manufacturing, Chongqing Technology and Business University（重庆工商大学智能制造国家研究基地）；School of Mechanical Engineering, Chongqing Technology and Business University（重庆工商大学机械工程学院）；School of Electrical and Mechanical Engineering, University of Adelaide（阿德莱德大学机电工程学院）
In this paper, design and development of a sensor integrated adaptive gripper is presented. Adaptive grippers are useful for grasping objects of varied geometric shapes by wrapping fingers around the object. The finger closing sequence in adaptive grippers may lead to ejection of the object from the gripper due to any unbalanced grasping force and such grasp failure is common for lightweight objects. Designing of the proposed gripper is focused on ensuring stable grasp on a wide variety of objects, especially, for lightweight objects (e.g., empty plastic bottles). The actuation mechanism is ba…
Safe yet stable grasping requires a robotic hand to apply sufficient force on the object to immobilize it while keeping it from getting damaged. Soft robotic hands have been proposed for safe grasping due to their passive compliance, but even such a hand can crush objects if the applied force is too high. Thus for safe grasping, regulating the grasping force is of uttermost importance even with soft hands. In this work, we present a force controlled soft hand and use it to achieve safe grasping. To this end, resistive force and bend sensors are integrated in a soft hand, and a data-driven cali…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook人工智能研究院）
Dexterous robotic hands are appealing for their agility and human-like morphology, yet their high degree of freedom makes learning to manipulate challenging. We introduce an approach for learning dexterous grasping. Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop to learn grasping policies that favor the same object regions favored by people. Unlike traditional approaches that learn from human demonstration trajectories (e.g., hand joint sequences captured with a glove), the proposed prior is object-centric and image-based, allowing …
