Shared control in teleoperation for providing robot assistance to accomplish object manipulation, called telemanipulation, is a new promising yet challenging problem. This has unique challenges--on top of teleoperation challenges in general--due to difficulties of physical discrepancy between human hands and robot hands as well as the fine motion constraints to constitute task success. We present an intuitive shared-control strategy where the focus is on generating robotic grasp poses which are better suited for human perception of successful teleoperated object manipulation and feeling of being in control of the robot, rather than developing objective stable grasp configurations for task success or following the human motion. The former is achieved by understanding human intent and autonomously taking over control on that inference. The latter is achieved by considering human inputs as hard motion constraints which the robot must abide. An arbitration of these two enables a trade-off for the subsequent robot motion to balance accomplishing the inferred task and motion constraints imposed by the operator. The arbitration framework adapts to the level of physical discrepancy between the human and different robot structures, enabling the assistance to indicate and appear to intuitively follow the user. To understand how users perceive good arbitration in object telemanipulation, we have conducted a user study with a hands-free telemanipulation setup to analyze the effect of factors including task predictability, perceived following, and user preference. The hands-free telemanipulation scene is chosen as the validation platform due to its more urgent need of intuitive robotics assistance for task success.
TASC: Task-Aware Shared Control for Relational Telemanipulation
KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics（KU莱顿机械工程系，机器人、自动化与机电一体化研究单位）；KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images（KU莱顿电气工程系，语音与图像处理研究单位）
Istituto Italiano di Tecnologia（意大利理工学院）；Universita di Pisa（比萨大学）
Teleoperation is a widely adopted strategy to control robotic manipulators executing complex tasks that require highly dexterous movements and critical high-level intelligence. Classical teleoperation schemes are based on either joystick control, or on more intuitive interfaces which map directly the user arm motions into one robot arm's motions. These approaches have limits when the execution of a given task requires reconfigurable multiple robotic arm systems. Indeed, the simultaneous teleoperation of two or more robot arms could extend the workspace of the manipulation cell, or increase its…
Colorado School of Mines（科罗拉多矿业学院）；Duke University（杜克大学）
In teleoperation, research has mainly focused on target approaching, where we deal with the more challenging object manipulation task by advancing the shared control technique. Appropriately manipulating an object is challenging due to the fine motion constraint requirements for a specific manipulation task. Although these motion constraints are critical for task success, they often are subtle when observing ambiguous human motion. The disembodiment problem and physical discrepancy between the human and robot hands bring additional uncertainty, further exaggerating the complications of the obj…
Legible and Intuitive Multi-modal Robot State and Intent Communication Validated in Online and Real-world Studies
Chair of Perception for Intelligent Systems, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM)（慕尼黑工业大学慕尼黑机器人与机器智能研究所智能系统感知教席）；Centre for Applied Autonomous Sensor Systems (AASS), Örebro University（厄勒布鲁大学应用自主传感器系统中心）；Robotics Institute Germany (RIG)（德国机器人研究所）
The rapid growth of e-commerce increases the need for larger warehouses and their automation, thus using robots as assistants to human workers becomes a priority. In order to operate efficiently and safely, robot assistants or the supervising system should recognize human intentions. Theory of mind (ToM) is an intuitive conception of other agents' mental state, i.e., beliefs and desires, and how they cause behavior. In this paper we present a ToM-based algorithm for human intention recognition in flexible robotized warehouses. We have placed the warehouse worker in a simulated 2D environment w…
The combination of collaborative robots and end-to-end AI, promises flexible automation of human tasks in factories and warehouses. However, such promise seems a few breakthroughs away. In the meantime, humans and cobots will collaborate helping each other. For these collaborations to be effective and safe, robots need to model, predict and exploit human's intents for responsive decision making processes. Approximate Bayesian Computation (ABC) is an analysis-by-synthesis approach to perform probabilistic predictions upon uncertain quantities. ABC includes priors conveniently, leverages samplin…
As the use of Augmented Reality (AR) to enhance interactions between human agents and robotic systems in a work environment continues to grow, robots must communicate their intents in informative yet straightforward ways. This improves the human agent's feeling of trust and safety in the work environment while also reducing task completion time. To this end, we discuss a set of guidelines for the systematic design of AR interfaces for Human-Robot Interaction (HRI) systems. Furthermore, we develop design frameworks that would ride on these guidelines and serve as a base for researchers seeking …
University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Laboratory for Autonomous Systems and Mobile Robotics（自主系统与移动机器人实验室）；Institute for Anthropomatics and Robotics（人机工程与机器人研究所）；Intelligent Process Automation and Robotics Lab（智能过程自动化与机器人实验室）
With the substantial growth of logistics businesses the need for larger warehouses and their automation arises, thus using robots as assistants to human workers is becoming a priority. In order to operate efficiently and safely, robot assistants or the supervising system should recognize human intentions in real-time. Theory of mind (ToM) is an intuitive human conception of other humans' mental state, i.e., beliefs and desires, and how they cause behavior. In this paper we propose a ToM based human intention estimation algorithm for flexible robotized warehouses. We observe human's, i.e., work…
The Use of Gaze-Derived Confidence of Inferred Operator Intent in Adjusting Safety-Conscious Haptic Assistance
Apple Inc.（苹果公司）；Perelman School of Medicine at University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）；Honda Research Institute US（美国本田研究所）；Colorado School of Mines（科罗拉多矿业学院）
Adaptive Motion Planning via Contact-Based Intent Inference for Human-Robot Collaboration
Department of Mechanical Engineering, Texas A&M University（机械工程系，德克萨斯A&M大学）；Department of Civil and Environmental Engineering, Texas A&M University（土木与环境工程系，德克萨斯A&M大学）
Human teams exhibit both implicit and explicit intention sharing. To further development of human-robot collaboration, intention recognition is crucial on both sides. Present approaches rely on a vast sensor suite on and around the robot to achieve intention recognition. This relegates intuitive human-robot collaboration purely to such bulky systems, which are inadequate for large-scale, real-world scenarios due to their complexity and cost. In this paper we propose an intention recognition system that is based purely on a portable head-mounted display. In addition robot intention visualisatio…
Tactile-Based Human Intent Recognition for Robot Assistive Navigation
Electrical & Computer Engineering, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校电气与计算机工程系）；Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院）
The functional and temporal roles of gaze evolve across the phases and constraints of multi-stage robot-mediated manipulation
The BioRobotics Institute, Scuola Superiore Sant’Anna（生物机器人研究所，高等圣安娜学院）；Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna（机器人与人工智能卓越部门，高等圣安娜学院）
Towards Intuitive Human-Robot Interaction through Embodied Gesture-Driven Control with Woven Tactile Skins
Centre for Perceptual and Interactive Intelligence (CPII)（感知与交互智能中心）；Department of Mechanical and Automation Engineering（机械与自动化工程系）；Texense Ltd.（Texense有限公司）；Department of Mechanical, Aerospace and Civil Engineering（机械、航空航天与土木工程系）
Huazhong University of Science and Technology（华中科技大学）；Nagoya University（名古屋大学）
Eye movement is closely related to limb actions, so it can be used to infer movement intentions. More importantly, in some cases, eye movement is the only way for paralyzed and impaired patients with severe movement disorders to communicate and interact with the environment. Despite this, eye-tracking technology still has very limited application scenarios as an intention recognition method. The goal of this paper is to achieve a natural fixation-based grasping intention recognition method, with which a user with hand movement disorders can intuitively express what tasks he/she wants to do by …
In order to provide therapy in a functional context, controls for wearable robotic orthoses need to be robust and intuitive. We have previously introduced an intuitive, user-driven, EMG-based method to operate a robotic hand orthosis, but the process of training a control that is robust to concept drift (changes in the input signal) places a substantial burden on the user. In this paper, we explore semi-supervised learning as a paradigm for controlling a powered hand orthosis for stroke subjects. To the best of our knowledge, this is the first use of semi-supervised learning for an orthotic ap…
Before the Body Moves: Learning Anticipatory Joint Intent for Language-Conditioned Humanoid Control
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；LimX Dynamics Technology Co., Ltd.（LimX动态技术有限公司）；Shandong University（山东大学）；Data61/CSIRO；Griffith University（格里菲斯大学）；Institute of Deep Perception Technology, Jiangsu Industrial Technology Research Institute (JITRI)（深度感知技术研究院，江苏省工业技术研究院（JITRI））
Humans engaged in collaborative activities are naturally able to convey their intentions to teammates through multi-modal communication, which is made up of explicit and implicit cues. Similarly, a more natural form of human-robot collaboration may be achieved by enabling robots to convey their intentions to human teammates via multiple communication channels. In this paper, we postulate that a better communication may take place should collaborative robots be able to anticipate their movements to human teammates in an intuitive way. In order to support such a claim, we propose a robot system'…
Advancements in the domain of physical human-robot interaction (pHRI) have tremendously improved the ability of humans and robots to communicate, collaborate, and coexist. In particular, compliant robotic systems offer many characteristics that can be leveraged towards enabling physical interactions that more efficiently and intuitively communicate intent, making compliant systems potentially useful in more physically demanding subsets of human-robot collaborative scenarios. Tensegrity robots are an example of compliant systems that are well-suited to physical interactions while still retainin…
TU Darmstadt（达姆施塔特工业大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；Goethe University（歌德大学）
Assistive robots can potentially improve the quality of life and personal independence of elderly people by supporting everyday life activities. To guarantee a safe and intuitive interaction between human and robot, human intentions need to be recognized automatically. As humans communicate their intentions multimodally, the use of multiple modalities for intention recognition may not just increase the robustness against failure of individual modalities but especially reduce the uncertainty about the intention to be predicted. This is desirable as particularly in direct interaction between rob…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of Michigan（密歇根大学）
Communication and cooperation among team members can be enhanced significantly with physical interaction. Successful collaboration requires the integration of the individual partners' intentions into a shared action plan, which may involve a continuous negotiation of intentions and roles. This paper presents an adaptive haptic shared control framework wherein a human driver and an automation system are physically connected through a motorized steering wheel. By virtue of haptic feedback, the driver and automation system can monitor each other actions and can still intuitively express their con…
In this study, an adaptive object deformability-agnostic human-robot collaborative transportation framework is presented. The proposed framework enables to combine the haptic information transferred through the object with the human kinematic information obtained from a motion capture system to generate reactive whole-body motions on a mobile collaborative robot. Furthermore, it allows rotating the objects in an intuitive and accurate way during co-transportation based on an algorithm that detects the human rotation intention using the torso and hand movements. First, we validate the framework…
Istituto Italiano di Tecnologia（意大利技术研究院）；Politecnico di Milano（米兰理工大学）
The recognition of actions performed by humans and the anticipation of their intentions are important enablers to yield sociable and successful collaboration in human-robot teams. Meanwhile, robots should have the capacity to deal with multiple objectives and constraints, arising from the collaborative task or the human. In this regard, we propose vision techniques to perform human action recognition and image classification, which are integrated into an Augmented Hierarchical Quadratic Programming (AHQP) scheme to hierarchically optimize the robot's reactive behavior and human ergonomics. The…
We propose a set of communicative gestures and develop a gesture recognition system with the aim of facilitating more intuitive Human-Robot Interaction (HRI) through gestures. First, we propose a set of commands commonly used for human-robot interaction. Next, an online user study with 190 participants was performed to investigate if there was an agreed set of gestures that people intuitively use to communicate the given commands to robots when no guidance or training were given. As we found large variations among the gestures exist between participants, we then proposed a set of gestures for …
Chinese Academy of Sciences（中国科学院）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）
Surface electromyogram (sEMG) is arguably the most sought-after physiological signal with a broad spectrum of biomedical applications, especially in miniaturized rehabilitation robots such as multifunctional prostheses. The widespread use of sEMG to drive pattern recognition (PR)-based control schemes is primarily due to its rich motor information content and non-invasiveness. Moreover, sEMG recordings exhibit non-linear and non-uniformity properties with inevitable interferences that distort intrinsic characteristics of the signal, precluding existing signal processing methods from yielding r…
Mixed Reality (MR) has recently shown great success as an intuitive interface for enabling end-users to teach robots. Related works have used MR interfaces to communicate robot intents and beliefs to a co-located human, as well as developed algorithms for taking multi-modal human input and learning complex motor behaviors. Even with these successes, enabling end-users to teach robots complex motor tasks still poses a challenge because end-user communication is highly task dependent and world knowledge is highly varied. We propose a learning framework where end-users teach robots a) motion demo…
University of Minnesota（明尼苏达大学）；University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；Nerves Incorporated（Nerves公司）；Fasikl Incorporated（Fasikl公司）
Objective: The next generation prosthetic hand that moves and feels like a real hand requires a robust neural interconnection between the human minds and machines. Methods: Here we present a neuroprosthetic system to demonstrate that principle by employing an artificial intelligence (AI) agent to translate the amputee's movement intent through a peripheral nerve interface. The AI agent is designed based on the recurrent neural network (RNN) and could simultaneously decode six degree-of-freedom (DOF) from multichannel nerve data in real-time. The decoder's performance is characterized in motor …
Wearable orthoses can function both as assistive devices, which allow the user to live independently, and as rehabilitation devices, which allow the user to regain use of an impaired limb. To be fully wearable, such devices must have intuitive controls, and to improve quality of life, the device should enable the user to perform Activities of Daily Living. In this context, we explore the feasibility of using electromyography (EMG) signals to control a wearable exotendon device to enable pick and place tasks. We use an easy to don, commodity forearm EMG band with 8 sensors to create an EMG patt…
AnyUser: Translating Sketched User Intent into Domestic Robots
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学国家机器人视觉感知与控制技术工程研究中心）
