Dexterous robotic manipulation remains a challenging domain due to its strict demands for precision and robustness on both hardware and software. While dexterous robotic hands have demonstrated remarkable capabilities in complex tasks, efficiently learning adaptive control policies for hands still presents a significant hurdle given the high dimensionalities of hands and tasks. To bridge this gap, we propose Tilde, an imitation learning-based in-hand manipulation system on a dexterous DeltaHand. It leverages 1) a low-cost, configurable, simple-to-control, soft dexterous robotic hand, DeltaHand, 2) a user-friendly, precise, real-time teleoperation interface, TeleHand, and 3) an efficient and generalizable imitation learning approach with diffusion policies. Our proposed TeleHand has a kinematic twin design to the DeltaHand that enables precise one-to-one joint control of the DeltaHand during teleoperation. This facilitates efficient high-quality data collection of human demonstrations in the real world. To evaluate the effectiveness of our system, we demonstrate the fully autonomous closed-loop deployment of diffusion policies learned from demonstrations across seven dexterous manipulation tasks with an average 90% success rate.
TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；Stanford University, USA（斯坦福大学）；Imperial College London, UK（伦敦帝国理工学院）
UniBiDex: A Unified Teleoperation Framework for Robotic Bimanual Dexterous Manipulation
Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Dongguan Key Laboratory of Intelligent Equipment and Smart Industry, School of Advanced Engineering, The Great Bay University（东莞智能装备与智能制造重点实验室，高级工程学院，大湾区大学）；Department of Mechanical Engineering, The Hong Kong Polytechnic University（香港理工大学机械工程系）；Guangdong Academy of Sciences（广东省科学院）
Universität Hamburg（汉堡大学）；Tsinghua University（清华大学）
In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulate…
This paper explores the possibility of improving bilateral robot manipulation task performance through optimizing the robot morphology and configuration of the system through motion. To optimize the design for different scenarios, we select a set of tasks that represent the variability in small scale manipulation (e.g. pick and place, tasks involving positioning and orientation) and track the motion to obtain a reproducible trajectory. Kinematic data is captured through an electromagnetic (EM) tracker system while a human subject performs the tasks. Then, the data is pre-processed and used to …
DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation
Autonomous Systems Lab, Institute of Computer Technology, TU Wien（自主系统实验室，计算机技术学院，维也纳技术大学）；Institute of Robotics and Mechatronics (DLR), German Aerospace Center（机器人与机电系统研究所（DLR），德国航空航天中心）
Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA
Shanghai Jiao Tong University（上海交通大学）；Sharpa；Shanghai AI Lab（上海人工智能实验室）；Zhongguancun Academy（中关村学院）；National University of Singapore（新加坡国立大学）；Shanghai Innovation Institute（上海创新研究院）
KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation
Tongji University（同济大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；University of Hong Kong（香港大学）；Shanghai Qi Zhi Institute（上海启智研究院）；Shanghai AI Lab（上海人工智能实验室）
Hunan University（湖南大学）；Bielefeld University（比勒费尔德大学）；University of the West of England（西英格兰大学）
In this work, we focus on improving the robot's dexterous capability by exploiting visual sensing and adaptive force control. TeachNet, a vision-based teleoperation learning framework, is exploited to map human hand postures to a multi-fingered robot hand. We augment TeachNet, which is originally based on an imprecise kinematic mapping and position-only servoing, with a biomimetic learning-based compliance control algorithm for dexterous manipulation tasks. This compliance controller takes the mapped robotic joint angles from TeachNet as the desired goal, computes the desired joint torques. It…
Hand-in-the-Loop: Improving VLA Policies for Dexterous Manipulation via Seamless Hand-Arm Intervention
State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University（机械系统与振动国家重点实验室，机械工程学院，上海交通大学）；Shanghai Key Laboratory of Intelligent Robotics, Meta Robotics Institute, Shanghai Jiao Tong University, Shanghai 200240, China（智能机器人上海市重点实验室，元机器人研究院，上海交通大学，上海200240，中国）；The University of Tokyo（东京大学）
In-hand pivoting is one of the important manipulation skills that leverage robot grippers' extrinsic dexterity to perform repositioning tasks to compensate for environmental uncertainties and imprecise motion execution. Although many researchers have been trying to solve pivoting problems using mathematical modeling or learning-based approaches, the problems remain as open challenges. On the other hand, humans perform in-hand manipulation with remarkable precision and speed. Hence, the solution could be provided by making full use of this intrinsic human skill through dexterous teleoperation. …
Analyzing Key Objectives in Human-to-Robot Retargeting for Dexterous Manipulation
Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, China（自动化系，信息科学与技术国家工程研究中心，清华大学）
Suction Leap-Hand: Suction Cups on a Multi-fingered Hand Enable Embodied Dexterity and In-Hand Teleoperation
School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；School of Physics, Engineering and Technology, University of York（物理、工程与技术学院，约克大学）；Tsinghua University（清华大学）
Remote robot manipulation with human control enables applications where safety and environmental constraints are adverse to humans (e.g. underwater, space robotics and disaster response) or the complexity of the task demands human-level cognition and dexterity (e.g. robotic surgery and manufacturing). These systems typically use direct teleoperation at the motion level, and are usually limited to low-DOF arms and 2D perception. Improving dexterity and situational awareness demands new interaction and planning workflows. We explore the use of human-robot teaming through teleautonomy with assist…
VR-DAgger: Immersive VR for Dexterous Data Collection and Uncertainty-Guided On-Policy Correction
Robotics Systems Lab（机器人系统实验室）；ETH Zürich（苏黎世联邦理工学院）；ETH AI Center（ETH人工智能中心）；ETH Augmented Reality Research Lab（ETH增强现实研究实验室）；ETH Mobility Initiative（ETH移动性倡议）；ANYbotics AG（ANYbotics公司）；Swiss Federal Railways（瑞士联邦铁路）
