Effective execution of long-horizon tasks with dexterous robotic hands remains a significant challenge in real-world problems. While learning from human demonstrations have shown encouraging results, they require extensive data collection for training. Hence, decomposing long-horizon tasks into reusable primitive skills is a more efficient approach. To achieve so, we developed DexSkills, a novel supervised learning framework that addresses long-horizon dexterous manipulation tasks using primitive skills. DexSkills is trained to recognize and replicate a select set of skills using human demonstration data, which can then segment a demonstrated long-horizon dexterous manipulation task into a sequence of primitive skills to achieve one-shot execution by the robot directly. Significantly, DexSkills operates solely on proprioceptive and tactile data, i.e., haptic data. Our real-world robotic experiments show that DexSkills can accurately segment skills, thereby enabling autonomous robot execution of a diverse range of tasks.
In-hand pivoting is one of the important manipulation skills that leverage robot grippers' extrinsic dexterity to perform repositioning tasks to compensate for environmental uncertainties and imprecise motion execution. Although many researchers have been trying to solve pivoting problems using mathematical modeling or learning-based approaches, the problems remain as open challenges. On the other hand, humans perform in-hand manipulation with remarkable precision and speed. Hence, the solution could be provided by making full use of this intrinsic human skill through dexterous teleoperation. …
Pohang University of Science and Technology（浦项科技大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Samsung research（三星研究院）
This paper addresses a data-driven framework, modeling expert driving skills for performance-based haptic assistance using neural networks (NNs). We have built a haptic driving training simulator to collect expert driving data and to provide proper haptic feedback. We establish an expert driving skill model by training NNs with the collected data. Then, the skill model is applied to performance-based haptic assistance to provide optimized references of the steering/pedaling movements. We evaluate the skill model and its application to performance-based haptic assistance in two user experiments…
In this paper, we propose a data-driven skill learning approach to solve highly dynamic manipulation tasks entirely from offline teleoperated play data. We use a bilateral teleoperation system to continuously collect a large set of dexterous and agile manipulation behaviors, which is enabled by providing direct force feedback to the operator. We jointly learn the state conditional latent skill distribution and skill decoder network in the form of goal-conditioned policy and skill conditional state transition dynamics using a two-stage generative modeling framework. This allows one to perform r…
People learn motor activities best when they are conscious of their errors and make a concerted effort to correct them. While haptic interfaces can facilitate motor training, existing interfaces are often bulky and do not always ensure post-training skill retention. Here, we describe a programmable haptic sleeve composed of textile-based electroadhesive clutches for skill acquisition and retention. We show its functionality in a motor learning study where users control a drone's movement using elbow joint rotation. Haptic feedback is used to restrain elbow motion and make users aware of their …
Skill-informed Data-driven Haptic Nudges for High-dimensional Human Motor Learning
Department of Electrical and Computer Engineering, Michigan State University（电气与计算机工程系，密歇根州立大学）；Department of Kinesiology and the Department of Mechanical Engineering, Michigan State University（运动科学系和机械工程系，密歇根州立大学）
LIRMM, University of Montpellier, CNRS（蒙彼利埃大学法国国家科学研究中心LIRMM实验室）；Stanford University（斯坦福大学）
The current generation of surgeons requires extensive training in teleoperation to develop specific dexterous skills, which are independent of medical knowledge. Training curricula progress from manipulation tasks to simulated surgical tasks but are limited in time. To tackle this, we propose to integrate surgical robotic training together with Haptic Feedback (HF) to improve skill acquisition. This paper present the initial but promising results of our haptic device designed to support in the training of surgical gestures. Our ongoing work is related to integrate the HF in the RAVEN II platfo…
Haptic exploration is a key skill for both robots and humans to discriminate and handle unknown objects or to recognize familiar objects. Its active nature is evident in humans who from early on reliably acquire sophisticated sensory-motor capabilities for active exploratory touch and directed manual exploration that associates surfaces and object properties with their spatial locations. This is in stark contrast to robotics. In this field, the relative lack of good real-world interaction models - along with very restricted sensors and a scarcity of suitable training data to leverage machine l…
Wrist-Squeezing Force Feedback Improves Accuracy and Speed in Robotic Surgery Training
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德州大学西南医学中心）
Surgical skill directly affects surgical procedure outcomes; thus, effective training is needed to ensure satisfactory results. Many objective assessment metrics have been developed and some are widely used in surgical training simulators. These objective metrics provide the trainee with descriptive feedback about their performance however, often lack feedback on how to proceed to improve performance. The most effective training method is one that is intuitive, easy to understand, personalized to the user and provided in a timely manner. We propose a framework to enable user-adaptive training …
CNRS-University of Montpellier（法国国家科学研究中心-蒙彼利埃大学）；CNRS-AIST Joint Robotics Laboratory（法国国家科学研究中心-产业技术综合研究所联合机器人实验室）
Bidirectional object handover between a human and a robot enables an important functionality skill in robotic human-centered manufacturing or services. The problem in achieving this skill lies in the capacity of any solution to deal with three important aspects: (i) synchronized timing for the handing over phases; (ii) the handling of object pose constraints; and (iii) understanding the haptic exchanging to seamlessly achieve some steps of the (i). We propose a new approach for (i) and (ii) consisting in explicitly formulating the handover process as constraints in a task-space quadratic progr…
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）
Cutting is a common form of manipulation when working with divisible objects such as food, rope, or clay. Cooking in particular relies heavily on cutting to divide food items into desired shapes. However, cutting food is a challenging task due to the wide range of material properties exhibited by food items. Due to this variability, the same cutting motions cannot be used for all food items. Sensations from contact events, e.g., when placing the knife on the food item, will also vary depending on the material properties, and the robot will need to adapt accordingly. In this paper, we propose u…
