TriFinger: An Open-Source Robot for Learning Dexterity
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）
Manuel Wüthrich, Felix Widmaier, Felix Grimminger, Joel Akpo, Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, J… 展开作者
Manuel Wüthrich, Felix Widmaier, Felix Grimminger, Joel Akpo, Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, Julian Viereck, Maximilien Naveau, Ludovic Righetti, Bernhard Schölkopf, Stefan Bauer
Dexterous object manipulation remains an open problem in robotics, despite the rapid progress in machine learning during the past decade. We argue that a hindrance is the high cost of experimentation on real systems, in terms of both time and money. We address this problem by proposing an open-source robotic platform which can safely operate without human supervision. The hardware is inexpensive (about \SI{5000}[\$]{}) yet highly dynamic, robust, and capable of complex interaction with external objects. The software operates at 1-kilohertz and performs safety checks to prevent the hardware from breaking. The easy-to-use front-end (in C++ and Python) is suitable for real-time control as well as deep reinforcement learning. In addition, the software framework is largely robot-agnostic and can hence be used independently of the hardware proposed herein. Finally, we illustrate the potential of the proposed platform through a number of experiments, including real-time optimal control, deep reinforcement learning from scratch, throwing, and writing.
A simple gripper can solve more complex manipulation tasks if it can utilize the external environment such as pushing the object against the table or a vertical wall, known as "Extrinsic Dexterity." Previous work in extrinsic dexterity usually has careful assumptions about contacts which impose restrictions on robot design, robot motions, and the variations of the physical parameters. In this work, we develop a system based on reinforcement learning (RL) to address these limitations. We study the task of "Occluded Grasping" which aims to grasp the object in configurations that are initially oc…
Tactile information plays a critical role in human dexterity. It reveals useful contact information that may not be inferred directly from vision. In fact, humans can even perform in-hand dexterous manipulation without using vision. Can we enable the same ability for the multi-finger robot hand? In this paper, we present Touch Dexterity, a new system that can perform in-hand object rotation using only touching without seeing the object. Instead of relying on precise tactile sensing in a small region, we introduce a new system design using dense binary force sensors (touch or no touch) overlayi…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Toyota Technological Institute（丰田技术研究院）
This report describes our winning submission to the Real Robot Challenge (https://real-robot-challenge.com/). The Real Robot Challenge is a three-phase dexterous manipulation competition that involves manipulating various rectangular objects with the TriFinger Platform. Our approach combines motion planning with several motion primitives to manipulate the object. For Phases 1 and 2, we additionally learn a residual policy in simulation that applies corrective actions on top of our controller. Our approach won first place in Phase 2 and Phase 3 of the competition. We were anonymously known as `…
Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Helmholtz and TU Munich（亥姆霍兹与慕尼黑工业大学）；Harvard University（哈佛大学）；DeepMind（深度思维）；University of Toronto（多伦多大学）；University College Dublin（都柏林大学学院）；Pohang University of Science and Technology（浦项科技大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）；ETH Zurich（苏黎世联邦理工学院）
本文介绍了 Real Robot Challenge 2022，该竞赛通过提供真实机器人离线数据集和远程评估平台，促进了强化学习与机器人社区的交流，并比较了获胜团队与最先进离线 RL 算法在灵巧操作任务上的表现。
University College Dublin（都柏林大学学院）；Dublin City University（都柏林城市大学）；Insight SFI Research Centre for Data Analytics（Insight SFI数据分析研究中心）；Harvard University（哈佛大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；KTH Stockholm（斯德哥尔摩皇家理工学院）
This paper describes a deep reinforcement learning (DRL) approach that won Phase 1 of the Real Robot Challenge (RRC) 2021, and then extends this method to a more difficult manipulation task. The RRC consisted of using a TriFinger robot to manipulate a cube along a specified positional trajectory, but with no requirement for the cube to have any specific orientation. We used a relatively simple reward function, a combination of goal-based sparse reward and distance reward, in conjunction with Hindsight Experience Replay (HER) to guide the learning of the DRL agent (Deep Deterministic Policy Gra…
Politecnico di Milano（米兰理工大学）；University College London（伦敦大学学院）
Robot-assisted laparoscopic prostatectomy (RALP) is a treatment for prostate cancer that involves complete or nerve sparing removal prostate tissue that contains cancer. After removal the bladder neck is successively sutured directly with the urethra. The procedure is called urethrovesical anastomosis and is one of the most dexterity demanding tasks during RALP. Two suturing instruments and a pair of needles are used in combination to perform a running stitch during urethrovesical anastomosis. While robotic instruments provide enhanced dexterity to perform the anastomosis, it is still highly c…
Dexterous Control of an 11-DOF Redundant Robot for CT-Guided Needle Insertion With Task-Oriented Weighted Policies
Jacobs School of Engineering, University of California San Diego（加州大学圣地亚哥分校 Jacobs 工程学院）；School of Medicine, University of California San Diego（加州大学圣地亚哥分校医学院）；School of Medicine, University of Missouri-Kansas City（密苏里大学堪萨斯城分校医学院）
Istituto Italiano di Tecnologia（意大利理工学院）；University of Genova（热那亚大学）；The University of Manchester（曼彻斯特大学）
Humanoid robot teleoperation allows humans to integrate their cognitive capabilities with the apparatus to perform tasks that need high strength, manoeuvrability and dexterity. This paper presents a framework for teleoperation of humanoid robots using a novel approach for motion retargeting through inverse kinematics over the robot model. The proposed method enhances scalability for retargeting, i.e., it allows teleoperating different robots by different human users with minimal changes to the proposed system. Our framework enables an intuitive and natural interaction between the human operato…
University of Southern California（南加州大学）；Harvard University（哈佛大学）
This work introduces an approach for automatic hair combing by a lightweight robot. For people living with limited mobility, dexterity, or chronic fatigue, combing hair is often a difficult task that negatively impacts personal routines. We propose a modular system for enabling general robot manipulators to assist with a hair-combing task. The system consists of three main components. The first component is the segmentation module, which segments the location of hair in space. The second component is the path planning module that proposes automatically-generated paths through hair based on use…
Continuum arms, such as trunk and tentacle robots, lie between the two extremities of rigid and soft robots and promise to capture the best of both worlds in terms of manipulability, dexterity, and compliance. This paper proposes a new kinematic model for a novel constant-length continuum robot that incorporates both soft and rigid elements. In contrast to traditional pneumatically actuated, variable-length continuum arms, the proposed design utilizes a hyper-redundant rigid chain to provide extra structural strength. The proposed model introduces a reduced-order mapping to account for mechani…
Teaching dexterity to multi-fingered robots has been a longstanding challenge in robotics. Most prominent work in this area focuses on learning controllers or policies that either operate on visual observations or state estimates derived from vision. However, such methods perform poorly on fine-grained manipulation tasks that require reasoning about contact forces or about objects occluded by the hand itself. In this work, we present T-Dex, a new approach for tactile-based dexterity, that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self…
Benchmarking Dexterity of Multifingered Robot Hands: A Review and Perspective
US National Science Foundation HAND Engineering Research Center（美国国家科学基金会HAND工程研究中心）；Center for Robotics and Biosystems, Northwestern Univ.（西北大学机器人与生物系统中心）；Department of Mechanical Engineering, Texas A&M Univ.（德克萨斯A&M大学机械工程系）；Department of Electrical and Computer Engineering, Carnegie Mellon Univ.（卡内基梅隆大学电子与计算机工程系）；Department of Mechanical Engineering, Florida A&M Univ.（佛罗里达A&M大学机械工程系）
SuFIA: Language-Guided Augmented Dexterity for Robotic Surgical Assistants
University of Toronto（多伦多大学）；University of Bern（伯尔尼大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）
Autonomous mobile manipulation offers a dual advantage of mobility provided by a mobile platform and dexterity afforded by the manipulator. In this paper, we present a whole-body optimal control framework to jointly solve the problems of manipulation, balancing and interaction as one optimization problem for an inherently unstable robot. The optimization is performed using a Model Predictive Control (MPC) approach; the optimal control problem is transcribed at the end-effector space, treating the position and orientation tasks in the MPC planner, and skillfully planning for end-effector contac…
The use of robots in minimally invasive surgery has improved the quality of standard surgical procedures. So far, only the automation of simple surgical actions has been investigated by researchers, while the execution of structured tasks requiring reasoning on the environment and the choice among multiple actions is still managed by human surgeons. In this paper, we propose a framework to implement surgical task automation. The framework consists of a task-level reasoning module based on answer set programming, a low-level motion planning module based on dynamic movement primitives, and a sit…
Combinatorial Optimization of Robotic Hand Kinematic Structures Using a Potential-Dexterity-Based QUBO Formulation
Advanced Robotics Research Center, Korea Institute of Machinery & Materials (KIMM)（韩国机械材料研究院先进机器人研究中心）；Mechanical Engineering, University of Science & Technology (UST)（科学技术大学机械工程系）
Stanford University（斯坦福大学）；Nvidia（英伟达）；University of Toronto（多伦多大学）
Large, richly annotated datasets have accelerated progress in fields such as computer vision and natural language processing, but replicating these successes in robotics has been challenging. While prior data collection methodologies such as self-supervision have resulted in large datasets, the data can have poor signal-to-noise ratio. By contrast, previous efforts to collect task demonstrations with humans provide better quality data, but they cannot reach the same data magnitude. Furthermore, neither approach places guarantees on the diversity of the data collected, in terms of solution stra…
Hong Kong University of Science and Technology（香港科技大学）
Adaptation to various scene configurations and object properties, stability and dexterity in robotic grasping manipulation is far from explored. This work presents an origami-based shape morphing fingertip design to actively tackle the grasping stability and dexterity problems. The proposed fingertip utilizes origami as its skeleton providing degrees of freedom at desired positions and motor-driven four-bar-linkages as its transmission components to achieve a compact size of the fingertip. 3 morphing types that are commonly observed and essential in robotic grasping are studied and validated w…
This paper develops closed-loop tactile controllers for dexterous robotic manipulation with a dual-palm robotic system. Tactile dexterity is an approach to dexterous manipulation that plans for robot/object interactions that render interpretable tactile information for control. We divide the role of tactile control into two goals: 1) control the contact state between the end-effector and the object (contact/no-contact, stick/slip) by regulating the stability of planned contact configurations and monitoring undesired slip events; and 2) control the object state by tactile-based tracking and ite…
Robot arm placements are oftentimes a limitation in surgical preoperative procedures, relying on trained staff to evaluate and decide on the optimal positions for the arms. Given new and different patient anatomies, it can be challenging to make an informed choice, leading to more frequently colliding arms or limited manipulator workspaces. In this paper, we develop a method to generate the optimal manipulator base positions for the multi-port da Vinci surgical system that minimizes self-collision and environment-collision, and maximizes the surgeon's reachability inside the patient. Scoring f…
To build general robotic agents that can operate in many environments, it is often imperative for the robot to collect experience in the real world. However, this is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real-world experience: internet videos of humans using their hands. Visual priors, such as visual features, are often learned from videos, but we believe that more information from videos can be utilized as a stronger prior. We build a learning algorithm, VideoDex, that leverages visual, action, and physical priors …
