Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；Nvidia（英伟达公司）；ETH Zurich（苏黎世联邦理工学院）；Snap（Snap公司）；MPI Tubingen（蒂宾根马克斯·普朗克研究所）；KTH（皇家理工学院）
Arthur Allshire, Mayank Mittal, Varun Lodaya, Viktor Makoviychuk, Denys Makoviichuk, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Ankur Handa, Animesh Garg
备注：
International Conference on Intelligent Robots and Systems (IROS 2022)
We present a system for learning a challenging dexterous manipulation task involving moving a cube to an arbitrary 6-DoF pose with only 3-fingers trained with NVIDIA's IsaacGym simulator. We show empirical benefits, both in simulation and sim-to-real transfer, of using keypoints as opposed to position+quaternion representations for the object pose in 6-DoF for policy observations and in reward calculation to train a model-free reinforcement learning agent. By utilizing domain randomization strategies along with the keypoint representation of the pose of the manipulated object, we achieve a high success rate of 83% on a remote TriFinger system maintained by the organizers of the Real Robot Challenge. With the aim of assisting further research in learning in-hand manipulation, we make the codebase of our system, along with trained checkpoints that come with billions of steps of experience available, at https://s2r2-ig.github.io
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）
Dexterous object manipulation remains an open problem in robotics, despite the rapid progress in machine learning during the past decade. We argue that a hindrance is the high cost of experimentation on real systems, in terms of both time and money. We address this problem by proposing an open-source robotic platform which can safely operate without human supervision. The hardware is inexpensive (about \SI{5000}[\$]{}) yet highly dynamic, robust, and capable of complex interaction with external objects. The software operates at 1-kilohertz and performs safety checks to prevent the hardware fro…
Technical University of Darmstadt（达姆施塔特工业大学）；University of Washington（华盛顿大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Cornell University（康奈尔大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Dexterous manipulation is a challenging and important problem in robotics. While data-driven methods are a promising approach, current benchmarks require simulation or extensive engineering support due to the sample inefficiency of popular methods. We present benchmarks for the TriFinger system, an open-source robotic platform for dexterous manipulation and the focus of the 2020 Real Robot Challenge. The benchmarked methods, which were successful in the challenge, can be generally described as structured policies, as they combine elements of classical robotics and modern policy optimization. T…
Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Helmholtz and TU Munich（亥姆霍兹与慕尼黑工业大学）；Harvard University（哈佛大学）；DeepMind（深度思维）；University of Toronto（多伦多大学）；University College Dublin（都柏林大学学院）；Pohang University of Science and Technology（浦项科技大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）；ETH Zurich（苏黎世联邦理工学院）
本文介绍了 Real Robot Challenge 2022，该竞赛通过提供真实机器人离线数据集和远程评估平台，促进了强化学习与机器人社区的交流，并比较了获胜团队与最先进离线 RL 算法在灵巧操作任务上的表现。
University College Dublin（都柏林大学学院）；Dublin City University（都柏林城市大学）；Insight SFI Research Centre for Data Analytics（Insight SFI数据分析研究中心）；Harvard University（哈佛大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；KTH Stockholm（斯德哥尔摩皇家理工学院）
This paper describes a deep reinforcement learning (DRL) approach that won Phase 1 of the Real Robot Challenge (RRC) 2021, and then extends this method to a more difficult manipulation task. The RRC consisted of using a TriFinger robot to manipulate a cube along a specified positional trajectory, but with no requirement for the cube to have any specific orientation. We used a relatively simple reward function, a combination of goal-based sparse reward and distance reward, in conjunction with Hindsight Experience Replay (HER) to guide the learning of the DRL agent (Deep Deterministic Policy Gra…
FoLD: Force-Informed Learning for Dexterous Articulated Object Manipulation
University of Science and Technology of China（中国科学技术大学）；Institute of AI for Industries, Chinese Academy of Sciences（中国科学院人工智能产业研究院）；Shenzhen University（深圳大学）
Analyzing Key Objectives in Human-to-Robot Retargeting for Dexterous Manipulation
Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, China（自动化系，信息科学与技术国家工程研究中心，清华大学）
DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation
Autonomous Systems Lab, Institute of Computer Technology, TU Wien（自主系统实验室，计算机技术学院，维也纳技术大学）；Institute of Robotics and Mechatronics (DLR), German Aerospace Center（机器人与机电系统研究所（DLR），德国航空航天中心）
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Toyota Technological Institute（丰田技术研究院）
This report describes our winning submission to the Real Robot Challenge (https://real-robot-challenge.com/). The Real Robot Challenge is a three-phase dexterous manipulation competition that involves manipulating various rectangular objects with the TriFinger Platform. Our approach combines motion planning with several motion primitives to manipulate the object. For Phases 1 and 2, we additionally learn a residual policy in simulation that applies corrective actions on top of our controller. Our approach won first place in Phase 2 and Phase 3 of the competition. We were anonymously known as `…
SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation
CFCS, School of Computer Science, Peking University（北京大学计算机学院CFCS）；Institute for AI, Peking University（北京大学人工智能研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；PKU-WUHAN Institute for Artificial Intelligence, China（北京大学武汉人工智能研究院）
Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Nanjing University（南京大学）；Central South University（中南大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Normal University（北京师范大学）
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This provides an intuitive interface and avoid unstable human-robot hand r…
Datasets are an essential component for training effective machine learning models. In particular, surgical robotic datasets have been key to many advances in semi-autonomous surgeries, skill assessment, and training. Simulated surgical environments can enhance the data collection process by making it faster, simpler and cheaper than real systems. In addition, combining data from multiple robotic domains can provide rich and diverse training data for transfer learning algorithms. In this paper, we present the DESK (Dexterous Surgical Skill) dataset. It comprises a set of surgical robotic skill…
D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping
Physical Superintelligence (PSI) Lab, University of Southern California（南加州大学物理超智能实验室）；Viterbi School of Engineering, University of Southern California（南加州大学韦伯尔工程学院）；Department of EECS, University of California, Berkeley（加州大学伯克利分校电子工程与计算机科学系）
