近期基于 3D 的操作方法要么直接使用 3D 神经网络预测抓取位姿,要么利用从形状数据库中检索到的相似物体求解抓取位姿。然而,前者在使用新机械臂或未见物体进行测试时面临泛化能力挑战;后者则假设数据库中存在相似物体。我们假设,近期的 3D 建模方法为构建评估场景的数字副本提供了一条途径,该副本能够支持物理仿真并支持鲁棒的操作算法学习。我们提出使用最先进的神经表面重建方法,从真实世界点云重建高质量网格(Real2Sim 步骤)。由于大多数仿真器使用网格进行快速仿真,重建的网格能够在无需人工投入的情况下生成抓取位姿标签。生成的标签可训练抓取网络,使其在真实评估场景中表现鲁棒(Sim2Real 步骤)。在合成实验和真实实验中,我们表明 Real2Sim2Real 流程的表现优于使用大型数据集训练的基线抓取网络,以及采用基于检索重建的抓取采样方法。Real2Sim2Real 流程的优势来自:1)将场景建模与抓取采样解耦为子问题;2)两个子问题都可以利用近期的 3D 学习算法和基于网格的物理仿真技术以足够高的质量得到解决。
英文摘要:
Recent 3D-based manipulation methods either directly predict the grasp pose using 3D neural networks, or solve the grasp pose using similar objects retrieved from shape databases. However, the former faces generalizability challenges when testing with new robot arms or unseen objects; and the latter assumes that similar objects exist in the databases. We hypothesize that recent 3D modeling methods provides a path towards building digital replica of the evaluation scene that affords physical simulation and supports robust manipulation algorithm learning. We propose to reconstruct high-quality meshes from real-world point clouds using state-of-the-art neural surface reconstruction method (the Real2Sim step). Because most simulators take meshes for fast simulation, the reconstructed meshes enable grasp pose labels generation without human efforts. The generated labels can train grasp network that performs robustly in the real evaluation scene (the Sim2Real step). In synthetic and real experiments, we show that the Real2Sim2Real pipeline performs better than baseline grasp networks trained with a large dataset and a grasp sampling method with retrieval-based reconstruction. The benefit of the Real2Sim2Real pipeline comes from 1) decoupling scene modeling and grasp sampling into sub-problems, and 2) both sub-problems can be solved with sufficiently high quality using recent 3D learning algorithms and mesh-based physical simulation techniques.
UltraDexGrasp: Learning Universal Dexterous Grasping for Bimanual Robots with Synthetic Data
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Zhejiang University(浙江大学);The University of Hong Kong(香港大学);Peking University(北京大学)
ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes
CFCS, School of Computer Science, Peking University(计算机学院,北京大学);PKU-AgiBot Lab(北京大学阿吉机器人实验室);PKU-PsiBot Lab(北京大学Psi机器人实验室);Princeton University(普林斯顿大学)
Designing Pin-pression Gripper and Learning its Dexterous Grasping with Online In-hand Adjustment
Dalian University of Technology(大连理工大学);Wuhan University(武汉大学);Shenyang University of Technology(沈阳理工大学);National University of Defense Technology(国防科技大学)
ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera
Samsung R&D Institute China-Beijing(三星中国北京研发中心);CFCS, School of Computer Science, Peking University(计算机科学系,北京大学);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院)
GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes
Korea Institute of Machinery & Materials (KIMM)(韩国机械材料研究院);Gwangju Institute of Science and Technology (GIST)(全州科学技术院);Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院)
GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priors
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Peking University(北京大学);The Hong Kong University of Science and Technology(香港科技大学)
Learning Robust Grasping Strategy Through Tactile Sensing and Adaption Skill
School of Power and Mechanical Engineering, Wuhan University(武汉大学动力与机械学院);School of Technological Sciences, Wuhan University(武汉大学技术科学学院);School of Microelectronics, Wuhan University(武汉大学微电子学院);School of Computer Science, Wuhan University(武汉大学计算机学院)
A Learning-Driven Framework with Spatial Optimization For Surgical Suture Thread Reconstruction and Autonomous Grasping Under Multiple Topologies and Environmental Noises
The Chinese University of Hong Kong(香港中文大学);Imperial College London(伦敦帝国学院)
Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success
Faculty of Electrical Engineering, Czech Technical University in Prague(捷克技术大学布拉格电子工程系);Czech Institute of Informatics, Robotics and Cybernetics(捷克信息学、机器人学与自动化研究所)
You Only Scan Once: A Dynamic Scene Reconstruction Pipeline for 6-DoF Robotic Grasping of Novel Objects
Advanced Robotics Centre, National University of Singapore(新加坡国立大学先进机器人中心);Integrative Sciences and Engineering Programme, National University of Singapore Graduate School(新加坡国立大学研究生院整合科学与工程项目)
GraspView: Active Perception Scoring and Best-View Optimization for Robotic Grasping in Cluttered Environments
Peng Cheng Laboratory(鹏城实验室);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院);Shenzhen Graduate School, Peking University(深圳研究生院);University of Chinese Academy of Sciences(中国科学院大学);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);X-Era AI Lab(X-Era AI实验室)
SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping
Beijing University of Posts and Telecommunications(北京邮电大学);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Triplane Grasping: Efficient 6-DoF Grasping with Single RGB Images
Computer Science Department, Faculty of Science and Engineering, Swansea University(计算机科学系,科学与工程学院,斯旺西大学);Department of Computer Science, Durham University(计算机科学系,达勒姆大学)
A Precise Real-Time Force-Aware Grasping System for Robust Aerial Manipulation
State Key Laboratory of Industrial Control Technology, Zhejiang University(浙江大学工业控制技术状态重点实验室);Huzhou Institute, Zhejiang University(浙江大学湖州研究院);State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University(浙江大学流体动力与机械系统状态重点实验室);Zhejiang Key Laboratory of Additive Manufacturing Technology and Equipment, School of Mechanical Engineering, Zhejiang University(浙江省增材制造技术与装备重点实验室)