In recent years, the need for robots to transition from isolated industrial tasks to shared environments, including human-robot collaboration and teleoperation, has become increasingly evident. Building on the foundation of Fractal Impedance Control (FIC) introduced in our previous work, this paper presents a novel extension to dual-arm tele-cooperation, leveraging the non-linear stiffness and passivity of FIC to adapt to diverse cooperative scenarios. Unlike traditional impedance controllers, our approach ensures stability without relying on energy tanks, as demonstrated in our prior research. In this paper, we further extend the FIC framework to bimanual operations, allowing for stable and smooth switching between different dynamic tasks without gain tuning. We also introduce a telemanipulation architecture that offers higher transparency and dexterity, addressing the challenges of signal latency and low-bandwidth communication. Through extensive experiments, we validate the robustness of our method and the results confirm the advantages of the FIC approach over traditional impedance controllers, showcasing its potential for applications in planetary exploration and other scenarios requiring dexterous telemanipulation. This paper's contributions include the seamless integration of FIC into multi-arm systems, the ability to perform robust interactions in highly variable environments, and the provision of a comprehensive comparison with competing approaches, thereby significantly enhancing the robustness and adaptability of robotic systems.
Grasp What You Want: Embodied Dexterous Grasping System Driven by Your Voice
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；South China University of Technology（华南理工大学）
DexTele: A Dual-Arm Dexterous Teleoperation System Based on Motion Retargeting and Adaptive Force Control
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（人工智能学院，香港中文大学（深圳））；School of Computing, University of Portsmouth（计算学院，朴茨茅斯大学）
Robotic teleoperation will allow us to perform complex manipulation tasks in dangerous or remote environments, such as needed for planetary exploration or nuclear decommissioning. This work proposes a novel telemanipulation architecture using a passive Fractal Impedance Controller (FIC), which does not depend upon an active viscous component for stability guarantees. Compared to a traditional impedance controller in ideal conditions (no delays and maximum communication bandwidth), our proposed method yields higher transparency in interaction and demonstrates superior dexterity and capability i…
This paper explores the possibility of improving bilateral robot manipulation task performance through optimizing the robot morphology and configuration of the system through motion. To optimize the design for different scenarios, we select a set of tasks that represent the variability in small scale manipulation (e.g. pick and place, tasks involving positioning and orientation) and track the motion to obtain a reproducible trajectory. Kinematic data is captured through an electromagnetic (EM) tracker system while a human subject performs the tasks. Then, the data is pre-processed and used to …
Universität Hamburg（汉堡大学）；Tsinghua University（清华大学）
In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulate…
UniBiDex: A Unified Teleoperation Framework for Robotic Bimanual Dexterous Manipulation
Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Dongguan Key Laboratory of Intelligent Equipment and Smart Industry, School of Advanced Engineering, The Great Bay University（东莞智能装备与智能制造重点实验室，高级工程学院，大湾区大学）；Department of Mechanical Engineering, The Hong Kong Polytechnic University（香港理工大学机械工程系）；Guangdong Academy of Sciences（广东省科学院）
CARIS: A Context-Adaptable Robot Interface System for Personalized and Scalable Human-Robot Interaction
Department of Systems and Computing Engineering, Mathematics, Universidad de los Andes（系统与计算工程系、数学系，Universidad de los Andes大学）；Department of Information Science, Cornell Tech（信息科学系，Cornell Tech）
Analyzing Key Objectives in Human-to-Robot Retargeting for Dexterous Manipulation
Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, China（自动化系，信息科学与技术国家工程研究中心，清华大学）
TeleOpBench: A Simulator-Centric Benchmark for Dual-Arm Dexterous Teleoperation
Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The University of Hong Kong（香港大学）；Feeling AI（FeelAI）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）
XRoboToolkit: A Cross-Platform Framework for Robot Teleoperation
ByteDance, PICO（字节跳动，PICO）；Georgia Institute of Technology, Institute for Robotics and Intelligent Machines (IRIM)（佐治亚理工学院，机器人与智能机器研究所）；George Mason University（乔治·马歇尔大学）
Human-in-the-loop telerobotic systems (HiLTS) are robotic tools designed to extend and in some circumstances improve the dexterous capabilities of the human operator in virtual and remote environments. Dexterous manipulation, however, depends on how well the telerobot is incorporated into the operator's sensorimotor control scheme. Empirical evidence suggests that haptic feedback can lead to improved dexterity. Unfortunately, haptic feedback can also introduce dynamics between the leader and follower of the telerobot that affect both stability and device performance. While concerted research e…
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This provides an intuitive interface and avoid unstable human-robot hand r…
DexCast: Learning Real-World Dexterous Control from Human Videos
Tsinghua University（清华大学）；TsingTec（清华科技园（注：TsingTec为清华大学相关科技产业机构，此处保留英文原名））；Fudan University（复旦大学）；Shenzhen University（深圳大学）；Nanyang Technological University（南洋理工大学）；Georgia Institute of Technology（佐治亚理工学院）
Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enables operators to carry out a variety of complex man…
Hand-in-the-Loop: Improving VLA Policies for Dexterous Manipulation via Seamless Hand-Arm Intervention
State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University（机械系统与振动国家重点实验室，机械工程学院，上海交通大学）；Shanghai Key Laboratory of Intelligent Robotics, Meta Robotics Institute, Shanghai Jiao Tong University, Shanghai 200240, China（智能机器人上海市重点实验室，元机器人研究院，上海交通大学，上海200240，中国）；The University of Tokyo（东京大学）
Physicians perform minimally invasive percutaneous procedures under Computed Tomography (CT) image guidance both for the diagnosis and treatment of numerous diseases. For these procedures performed within Computed Tomography Scanners, robots can enable physicians to more accurately target sub-dermal lesions while increasing safety. However, existing robots for this application have limited dexterity, workspace, or accuracy. This paper describes the design, manufacture, and performance of a highly dexterous, low-profile, 8+2 Degree-ofFreedom (DoF) robotic arm for CT guided percutaneous needle b…
This paper describes the design, manufacture, and performance of a highly dexterous, low-profile, 7 Degree-of-Freedom (DOF) robotic arm for CT-guided percutaneous needle biopsy. Direct CT guidance allows physicians to localize tumours quickly; however, needle insertion is still performed by hand. This system is mounted to a fully active gantry superior to the patient's head and teleoperated by a radiologist. Unlike other similar robots, this robot's fully serial-link approach uses a unique combination of belt and cable drives for high-transparency and minimal-backlash, allowing for an expansiv…
Achieving Dexterous Bidirectional Interaction in Uncertain Conditions for Medical Robotics
Institute for Perception, Action, and Behaviour, School of Informatics, University of Edinburgh（爱丁堡大学信息学院感知、行动与行为研究所）；University College London（伦敦大学学院）；University of Sussex（萨塞克斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Université de Lorraine（洛林大学）
Hunan University（湖南大学）；Bielefeld University（比勒费尔德大学）；University of the West of England（西英格兰大学）
In this work, we focus on improving the robot's dexterous capability by exploiting visual sensing and adaptive force control. TeachNet, a vision-based teleoperation learning framework, is exploited to map human hand postures to a multi-fingered robot hand. We augment TeachNet, which is originally based on an imprecise kinematic mapping and position-only servoing, with a biomimetic learning-based compliance control algorithm for dexterous manipulation tasks. This compliance controller takes the mapped robotic joint angles from TeachNet as the desired goal, computes the desired joint torques. It…
