Shared-autonomy imitation learning lets a human correct a robot in real time, mitigating covariate-shift errors. Yet existing approaches ignore two critical factors: (i) the operator's cognitive load and (ii) the risk created by delayed or erroneous interventions. We present an uncertainty-aware shared-autonomy system in which the robot modulates its behaviour according to a learned estimate of latent-space skill uncertainty. A hierarchical policy first infers a conservative skill embedding and then decodes it into low-level actions, enabling rapid task execution while automatically slowing down when uncertainty is high. We detail a full, open-source VR-teleoperation pipeline that is compatible with multi-configuration manipulators such as UR-series arms. Experiments on pouring and pick-and-place tasks demonstrate 70-90% success in dynamic scenes with moving targets, and a qualitative study shows a marked reduction in collision events compared with a non-conservative baseline. Although a dedicated ablation that isolates uncertainty is impractical on hardware for safety and cost reasons, the reported gains in stability and operator workload already validate the design and motivate future large-scale studies.
From Novice to Skilled: RL-based Shared Autonomy Communicating with Pilots in UAV Multi-Task Missions
Institute for Clarity in Documentation（文献清晰度研究所）；The Thørväld Group（索瓦尔德集团）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗康校区）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）；The Kumquat Consortium（金桔联盟）
Wheelchair-mounted robotic arms (and other assistive robots) should help their users perform everyday tasks. One way robots can provide this assistance is shared autonomy. Within shared autonomy, both the human and robot maintain control over the robot's motion: as the robot becomes confident it understands what the human wants, it increasingly intervenes to automate the task. But how does the robot know what tasks the human may want to perform in the first place? Today's shared autonomy approaches often rely on prior knowledge: for example, the robot must know the set of possible human goals …
Shared autonomy enables robots to infer user intent and assist in accomplishing it. But when the user wants to do a new task that the robot does not know about, shared autonomy will hinder their performance by attempting to assist them with something that is not their intent. Our key idea is that the robot can detect when its repertoire of intents is insufficient to explain the user's input, and give them back control. This then enables the robot to observe unhindered task execution, learn the new intent behind it, and add it to this repertoire. We demonstrate with both a case study and a user…
Failure-Aware Bimanual Teleoperation via Conservative Value Guided Assistance
School of Advanced Engineering, Great Bay University（先进工程学院，大湾大学）；School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Department of Mechanical Engineering, The Hong Kong Polytechnic University（机械工程系，香港理工大学）；College of Mechanical and Electrical Engineering, Northeast Forestry University（机械电子工程学院，东北林业大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电气与电子工程学院，南洋理工大学）
Northwestern University（西北大学）；Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）
Assistive robotic devices can increase the independence of individuals with motor impairments. However, each person is unique in their level of injury, preferences, and skills, which moreover can change over time. Further, the amount of assistance required can vary throughout the day due to pain or fatigue, or over longer periods due to rehabilitation, debilitating conditions, or aging. Therefore, in order to become an effective team member, the assistive machine should be able to learn from and adapt to the human user. To do so, we need to be able to characterize the user's control commands t…
University of California, Berkeley（加州大学伯克利分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；Massachusetts Institute of Technology（麻省理工学院）；Computing Community Consortium（计算社区联盟）
An Autonomous Physical System (APS) will be expected to reliably and independently evaluate, execute, and achieve goals while respecting surrounding rules, laws, or conventions. In doing so, an APS must rely on a broad spectrum of dynamic, complex, and often imprecise information about its surroundings, the task it is to perform, and its own sensors and actuators. For example, cleaning in a home or commercial setting requires the ability to perceive, grasp, and manipulate many physical objects, the ability to reliably perform a variety of subtasks such as washing, folding, and stacking, and kn…
Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration
Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）
Autonomous surgery involves having surgical tasks performed by a robot operating under its own will, with partial or no human involvement. There are several important advantages of automation in surgery, which include increasing precision of care due to sub-millimeter robot control, real-time utilization of biosignals for interventional care, improvements to surgical efficiency and execution, and computer-aided guidance under various medical imaging and sensing modalities. While these methods may displace some tasks of surgical teams and individual surgeons, they also present new capabilities …
Recent works on shared autonomy and assistive-AI technologies, such as assistive robot teleoperation, seek to model and help human users with limited ability in a fixed task. However, these approaches often fail to account for humans' ability to adapt and eventually learn how to execute a control task themselves. Furthermore, in applications where it may be desirable for a human to intervene, these methods may inhibit their ability to learn how to succeed with full self-control. In this paper, we focus on the problem of assistive teaching of motor control tasks such as parking a car or landing…
To enable robots to achieve high level objectives, engineers typically write scripts that apply existing specialized skills, such as navigation, object detection and manipulation to achieve these goals. Writing good scripts is challenging since they must intelligently balance the inherent stochasticity of a physical robot's actions and sensors, and the limited information it has. In principle, AI planning can be used to address this challenge and generate good behavior policies automatically. But this requires passing three hurdles. First, the AI must understand each skill's impact on the worl…
SharedAssembly: A Data Collection Approach via Shared Tele-Assembly
Technical University of Munich（慕尼黑工业大学）；Munich Institute of Robotics and Machine Intelligence（慕尼黑机器人与机器智能研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Centre for Perceptual and Interactive Intelligence（感知与交互智能中心）；ETH Zurich（苏黎世联邦理工学院）；University of Edinburgh（爱丁堡大学）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）
Human-AI shared control allows human to interact and collaborate with AI to accomplish control tasks in complex environments. Previous Reinforcement Learning (RL) methods attempt the goal-conditioned design to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm. Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called \textit{Policy Dissection} to align the intermediate representation of the learned neural controller with the kinematic attributes of t…
Catch, Throw, Repeat: Planning for Human-Robot Partner Juggling
Technical University of Darmstadt（达姆施塔特工业大学）；Justus-Liebig University Gießen（吉森尤斯 - 李比希大学）；German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI））；Robotics Institute Germany（德国机器人研究所）；Indian Institute of Technology Delhi - Abu Dhabi（印度理工学院德里分校 - 阿布扎比）
Learning a robot motor skill from scratch is impractically slow; so much so that in practice, learning must be bootstrapped using a good skill policy obtained from human demonstration. However, relying on human demonstration necessarily degrades the autonomy of robots that must learn a wide variety of skills over their operational lifetimes. We propose using kinematic motion planning as a completely autonomous, sample efficient way to bootstrap motor skill learning for object manipulation. We demonstrate the use of motion planners to bootstrap motor skills in two complex object manipulation sc…
Purdue University（普渡大学）；Indiana University School of Medicine（印第安纳大学医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Short response time is critical for future military medical operations in austere settings or remote areas. Such effective patient care at the point of injury can greatly benefit from the integration of semi-autonomous robotic systems. To achieve autonomy, robots would require massive libraries of maneuvers. While this is possible in controlled settings, obtaining surgical data in austere settings can be difficult. Hence, in this paper, we present the Dexterous Surgical Skill (DESK) database for knowledge transfer between robots. The peg transfer task was selected as it is one of 6 main tasks …
Robot manipulation is increasingly poised to interact with humans in co-shared workspaces. Despite increasingly robust manipulation and control algorithms, failure modes continue to exist whenever models do not capture the dynamics of the unstructured environment. To obtain longer-term horizons in robot automation, robots must develop introspection and recovery abilities. We contribute a set of recovery policies to deal with anomalies produced by external disturbances as well as anomaly classification through the use of non-parametric statistics with memoized variational inference with scalabl…
The Autonomy-Alignment Problem in Open-Ended Learning Robots: Formalising the Purpose Framework
Institute of Cognitive Sciences and Technologies, National Research Council（认知科学与技术研究所，国家研究理事会）；Integrated Group for Engineering Research, CITIC, Universidade da Coruña（工程研究联合组，CITIC，科鲁纳大学）；Institute of Intelligent Systems and Robotics, Sorbonne University / CNRS（智能系统与机器人研究所，索邦大学 / CNRS）
AssistDLO: Assistive Teleoperation for Deformable Linear Object Manipulation
TU Darmstadt, Institute for Intelligent Autonomous Systems, Computer Science Department（德累斯顿技术大学智能自主系统研究所，计算机科学系）；Honda Research Institute Europe GmbH（本田欧洲研究院）；German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning（德国人工智能研究中心（DFKI），研究部门：机器人学习的系统AI；黑森AI；德国机器人研究所（RIG）；认知科学中心）；Hessian.AI；Robotics Institute Germany (RIG)；Centre for Cognitive Science
Robotic Grasping has always been an active topic in robotics since grasping is one of the fundamental but most challenging skills of robots. It demands the coordination of robotic perception, planning, and control for robustness and intelligence. However, current solutions are still far behind humans, especially when confronting unstructured scenarios. In this paper, we survey the advances of robotic grasping, starting from the classical formulations and solutions to the modern ones. By reviewing the history of robotic grasping, we want to provide a complete view of this community, and perhaps…
Learning visuomotor control policies in robotic systems is a fundamental problem when aiming for long-term behavioral autonomy. Recent supervised-learning-based vision and motion perception systems, however, are often separately built with limited capabilities, while being restricted to few behavioral skills such as passive visual odometry (VO) or mobile robot visual localization. Here we propose an approach to unify those successful robot perception systems for active target-driven navigation tasks via reinforcement learning (RL). Our method temporally incorporates compact motion and visual p…
University of Cambridge（剑桥大学）；Google DeepMind（谷歌DeepMind）；UC Berkeley（加州大学伯克利分校）
Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinf…
Northwestern University（西北大学）；Shirley Ryan AbilityLab（雪莉·瑞恩能力实验室）
We describe a shared control methodology that can, without knowledge of the task, be used to improve a human's control of a dynamic system, be used as a training mechanism, and be used in conjunction with Imitation Learning to generate autonomous policies that recreate novel behaviors. Our algorithm introduces autonomy that assists the human partner by enforcing safety and stability constraints. The autonomous agent has no a priori knowledge of the desired task and therefore only adds control information when there is concern for the safety of the system. We evaluate the efficacy of our approa…
Learning to effectively imitate human teleoperators, with generalization to unseen and dynamic environments, is a promising path to greater autonomy enabling robots to steadily acquire complex skills from supervision. We propose a new motion learning technique rooted in contraction theory and sum-of-squares programming for estimating a control law in the form of a polynomial vector field from a given set of demonstrations. Notably, this vector field is provably optimal for the problem of minimizing imitation loss while providing continuous-time guarantees on the induced imitation behavior. Our…
Prompt2Auto: From Motion Prompt to Automated Control via Geometry-Invariant One-Shot Gaussian Process Learning
Chair of Robotics and Systems Intelligence, Munich Institute of Robotics and Machine Intelligence, Technical University of Munich（机器人与系统智能教授团，慕尼黑机器人与机器智能研究所，慕尼黑技术大学）；Chair of Information-oriented Control, TUM School of Computation, Information and Technology, Technical University of Munich（信息导向控制教授团，技术大学计算、信息与技术学院，慕尼黑技术大学）；Dalian University of Technology（大连理工大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Scientific Exploration of Challenging Planetary Analog Environments with a Team of Legged Robots
ETH Zurich（苏黎世联邦理工学院）；Harbin Institute of Technology（哈尔滨工业大学）；University of Bern（伯尔尼大学）；University of Basel（巴塞尔大学）；University of Zurich（苏黎世大学）；Lucerne University of Applied Sciences and Art（卢塞恩应用科学与艺术大学）
