Failure-Aware Bimanual Teleoperation via Conservative Value Guided Assistance
School of Advanced Engineering, Great Bay University（先进工程学院，大湾大学）；School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Department of Mechanical Engineering, The Hong Kong Polytechnic University（机械工程系，香港理工大学）；College of Mechanical and Electrical Engineering, Northeast Forestry University（机械电子工程学院，东北林业大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电气与电子工程学院，南洋理工大学）
Peng Zhou, Zhongxuan Li, Jinsong Wu, Jiaming Qi, Jun Hu, David Navarro-Alarcon, Jia Pan, Lihua Xie, Shiyao Zhang, Zeqing Zhang
Teleoperation of high-precision manipulation is con-strained by tight success tolerances and complex contact dy-namics, which make impending failures difficult for human operators to anticipate under partial observability. This paper proposes a value-guided, failure-aware framework for bimanual teleoperation that provides compliant haptic assistance while pre-serving continuous human authority. The framework is trained entirely from heterogeneous offline teleoperation data containing both successful and failed executions. Task feasibility is mod-eled as a conservative success score learned via Conservative Value Learning, yielding a risk-sensitive estimate that remains reliable under distribution shift. During online operation, the learned success score regulates the level of assistance, while a learned actor provides a corrective motion direction. Both are integrated through a joint-space impedance interface on the master side, yielding continuous guidance that steers the operator away from failure-prone actions without overriding intent. Experimental results on contact-rich manipulation tasks demonstrate improved task success rates and reduced operator workload compared to conventional teleoperation and shared-autonomy baselines, indicating that conservative value learning provides an effective mechanism for embedding failure awareness into bilateral teleoperation. Experimental videos are available at https://www.youtube.com/watch?v=XDTsvzEkDRE
Robotic assistance in robot arm teleoperation tasks has recently gained a lot of traction in industrial and domestic environment. A wide variety of input devices is used in such setups. Due to the noise in the input signals (e.g., Brain Computer Interface (BCI)) or delays due to environmental conditions (e.g., space robot teleoperation), users need assistive autonomy that keeps them in control while following predefined trajectories and avoids obstacles. This assistance calls for activity representations that are easy to define by the operator and able to take the dynamic world state into cons…
Carnegie Mellon University（卡内基梅隆大学）；National Research Council of Italy（意大利国家研究委员会）
In shared autonomy, a user and autonomous system work together to achieve shared goals. To collaborate effectively, the autonomous system must know the user's goal. As such, most prior works follow a predict-then-act model, first predicting the user's goal with high confidence, then assisting given that goal. Unfortunately, confidently predicting the user's goal may not be possible until they have nearly achieved it, causing predict-then-act methods to provide little assistance. However, the system can often provide useful assistance even when confidence for any single goal is low (e.g. move t…
Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh（匹兹堡大学）；Center for the Neural Basis of Cognition（认知神经基础中心）；VA Pittsburgh Healthcare System（匹兹堡退伍军人医疗保健系统）
Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain-Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the…
Northwestern University（西北大学）；Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）
Assistive robotic devices can increase the independence of individuals with motor impairments. However, each person is unique in their level of injury, preferences, and skills, which moreover can change over time. Further, the amount of assistance required can vary throughout the day due to pain or fatigue, or over longer periods due to rehabilitation, debilitating conditions, or aging. Therefore, in order to become an effective team member, the assistive machine should be able to learn from and adapt to the human user. To do so, we need to be able to characterize the user's control commands t…
UniBiDex: A Unified Teleoperation Framework for Robotic Bimanual Dexterous Manipulation
Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Dongguan Key Laboratory of Intelligent Equipment and Smart Industry, School of Advanced Engineering, The Great Bay University（东莞智能装备与智能制造重点实验室，高级工程学院，大湾区大学）；Department of Mechanical Engineering, The Hong Kong Polytechnic University（香港理工大学机械工程系）；Guangdong Academy of Sciences（广东省科学院）
Shared autonomy teleoperation can guarantee safety, but does so by reducing the human operator's control authority, which can lead to reduced levels of human-robot agreement and user satisfaction. This paper presents a novel haptic shared autonomy teleoperation paradigm that uses haptic feedback to inform the user about the inner state of a shared autonomy paradigm, while still guaranteeing safety. This differs from haptic shared control, which uses haptic feedback to inform the user's actions, but gives the human operator full control over the robot's actions. We conducted a user study in whi…
Northwestern University（西北大学）；Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）
We present an assistance system that reasons about a human's intended actions during robot teleoperation in order to provide appropriate corrections for unintended behavior. We model the human's physical interaction with a control interface during robot teleoperation and distinguish between intended and measured physical actions explicitly. By reasoning over the unobserved intentions using model-based inference techniques, our assistive system provides customized corrections on a user's issued commands. We validate our algorithm with a 10-person human subject study in which we evaluate the per…
Taming VR Teleoperation and Learning from Demonstration for Multi-Task Bimanual Table Service Manipulation
AIR, Tsinghua University（空气动力学研究所，清华大学）；Northwestern University（西北大学）；Georgia Institute of Technology（佐治亚理工学院）；HKUST (GZ)（香港科技大学（广州））；NUS (Suzhou) Research Institute（国立大学（苏州）研究所）；University of Michigan（密歇根大学）
Learning Sim-Grounded Policies for Bimanual Rope Manipulation from Human Teleoperation Data
Technical University of Darmstadt（德累斯顿技术大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Robotics Institute Germany (RIG)（德国机器人研究所）；Centre for Cognitive Science（认知科学研究中心）；Honda Research Institute Europe GmbH（本田欧洲研究院）
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting…
UC Berkeley（加州大学伯克利分校）；Microsoft Research（微软研究院）；University of Hertfordshire（赫特福德大学）
One difficulty in using artificial agents for human-assistive applications lies in the challenge of accurately assisting with a person's goal(s). Existing methods tend to rely on inferring the human's goal, which is challenging when there are many potential goals or when the set of candidate goals is difficult to identify. We propose a new paradigm for assistance by instead increasing the human's ability to control their environment, and formalize this approach by augmenting reinforcement learning with human empowerment. This task-agnostic objective preserves the person's autonomy and ability …
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting…
Confidence-based Intent Prediction for Teleoperation in Bimanual Robotic Suturing
Department of Mechanical Engineering, Imperial College London（机械工程系，帝国理工学院伦敦分校）；Department of Surgery & Cancer, Imperial College London（外科与癌症系，帝国理工学院伦敦分校）；Department of Engineering and Design, University of Sussex（工程与设计系，Sussex大学）；Department of Bioengineering, Imperial College London（生物工程系，帝国理工学院伦敦分校）
University of Waterloo（滑铁卢大学）；Delft University of Technology（代尔夫特理工大学）
In this paper, we consider the problem of allocating human operator assistance in a system with multiple autonomous robots. Each robot is required to complete independent missions, each defined as a sequence of tasks. While executing a task, a robot can either operate autonomously or be teleoperated by the human operator to complete the task at a faster rate. We show that the problem of creating a teleoperation schedule that minimizes makespan of the system is NP-Hard. We formulate our problem as a Mixed Integer Linear Program, which can be used to optimally solve small to moderate sized probl…
A Shared-Autonomy Construction Robotic System for Overhead Works
Intelligent Robotics Research Center, Korea Electronics Technology Institute (KETI)（韩国电子技术研究所智能机器人研究中心）；Dept. of Electrical and Computer Engineering, University of California, San Diego (UCSD)（加州大学圣地亚哥分校电气与计算机工程系）；ITONE Inc.（ITONE公司）
Assistive robots enable people with disabilities to conduct everyday tasks on their own. However, these tasks can be complex, containing both coarse reaching motions and fine-grained manipulation. For example, when eating, not only does one need to move to the correct food item, but they must also precisely manipulate the food in different ways (e.g., cutting, stabbing, scooping). Shared autonomy methods make robot teleoperation safer and more precise by arbitrating user inputs with robot controls. However, these works have focused mainly on the high-level task of reaching a goal from a discre…
Feasibility Retargeting for Multi-contact Teleoperation and Physical Interaction
Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Université de Lorraine（洛林大学）；University of Edinburgh（爱丁堡大学）；University College London（伦敦大学学院）；University of Sussex（萨塞克斯大学）
AssistDLO: Assistive Teleoperation for Deformable Linear Object Manipulation
TU Darmstadt, Institute for Intelligent Autonomous Systems, Computer Science Department（德累斯顿技术大学智能自主系统研究所，计算机科学系）；Honda Research Institute Europe GmbH（本田欧洲研究院）；German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning（德国人工智能研究中心（DFKI），研究部门：机器人学习的系统AI；黑森AI；德国机器人研究所（RIG）；认知科学中心）；Hessian.AI；Robotics Institute Germany (RIG)；Centre for Cognitive Science
