Transfer of objects between humans and robots is a critical capability for collaborative robots. Although there has been a recent surge of interest in human-robot handovers, most prior research focus on robot-to-human handovers. Further, work on the equally critical human-to-robot handovers often assumes humans can place the object in the robot's gripper. In this paper, we propose an approach for human-to-robot handovers in which the robot meets the human halfway, by classifying the human's grasp of the object and quickly planning a trajectory accordingly to take the object from the human's hand according to their intent. To do this, we collect a human grasp dataset which covers typical ways of holding objects with various hand shapes and poses, and learn a deep model on this dataset to classify the hand grasps into one of these categories. We present a planning and execution approach that takes the object from the human hand according to the detected grasp and hand position, and replans as necessary when the handover is interrupted. Through a systematic evaluation, we demonstrate that our system results in more fluent handovers versus two baselines. We also present findings from a user study (N = 9) demonstrating the effectiveness and usability of our approach with naive users in different scenarios. More results and videos can be found at http://wyang.me/handovers.
Human-robot object handovers have been an actively studied area of robotics over the past decade; however, very few techniques and systems have addressed the challenge of handing over diverse objects with arbitrary appearance, size, shape, and rigidity. In this paper, we present a vision-based system that enables reactive human-to-robot handovers of unknown objects. Our approach combines closed-loop motion planning with real-time, temporally-consistent grasp generation to ensure reactivity and motion smoothness. Our system is robust to different object positions and orientations, and can grasp…
Adaptive vs. Static Robot-to-Human Handover: A Study on Orientation and Approach Direction
Department of Engineering "Enzo Ferrari", University of Modena and Reggio Emilia（工程学院"Enzo Ferrari"，摩德纳和雷吉奥艾米利亚大学）；Department of Surgery, Medicine, Dentistry and Morphological Sciences, University of Modena and Reggio Emilia（外科、医学、牙科和形态学科学学院，摩德纳和雷吉奥艾米利亚大学）
Human-robot handover is a fundamental yet challenging task in human-robot interaction and collaboration. Recently, remarkable progressions have been made in human-to-robot handovers of unknown objects by using learning-based grasp generators. However, how to responsively generate smooth motions to take an object from a human is still an open question. Specifically, planning motions that take human comfort into account is not a part of the human-robot handover process in most prior works. In this paper, we propose to generate smooth motions via an efficient model-predictive control (MPC) framew…
Object handover is a common human collaboration behavior that attracts attention from researchers in Robotics and Cognitive Science. Though visual perception plays an important role in the object handover task, the whole handover process has been specifically explored. In this work, we propose a novel rich-annotated dataset, H2O, for visual analysis of human-human object handovers. The H2O, which contains 18K video clips involving 15 people who hand over 30 objects to each other, is a multi-purpose benchmark. It can support several vision-based tasks, from which, we specifically provide a base…
Expressing and Inferring Action Carefulness in Human-to-Robot Handovers
Italian Institute of Technology（意大利技术研究院）；University of Genoa（热那亚大学）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Institute for Systems and Robotics (ISR Lisboa)（系统与机器人研究所（里斯本））
NVIDIA（英伟达）；UT Dallas（德克萨斯大学达拉斯分校）；MIT（麻省理工学院）；University of Washington（华盛顿大学）
We introduce a new simulation benchmark "HandoverSim" for human-to-robot object handovers. To simulate the giver's motion, we leverage a recent motion capture dataset of hand grasping of objects. We create training and evaluation environments for the receiver with standardized protocols and metrics. We analyze the performance of a set of baselines and show a correlation with a real-world evaluation. Code is open sourced at https://handover-sim.github.io.
Generative Grasp Detection and Estimation with Concept Learning-based Safety Criteria
Innovative Factory Solutions（创新工厂解决方案）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Chair of Machine Tools and Control Systems (WSKL)（机械工具与控制系统教授职位）；Rheinland-Pfälzische Technische Universität (RPTU)（莱茵-瓦尔德技术大学）
R2HandoverSim: A Simulation Framework and Benchmark for Robot-to-Human Object Handovers
DANiLab, University of Leicester（莱斯特大学DANiLab）；School of Computing and Mathematical Sciences, University of Leicester（莱斯特大学计算与数学科学学院）；School of Metallurgy and Materials, University of Birmingham（伯明翰大学冶金与材料学院）
Humans display the remarkable ability to sense the world through tools and other held objects. For example, we are able to pinpoint impact locations on a held rod and tell apart different textures using a rigid probe. In this work, we consider how we can enable robots to have a similar capacity, i.e., to embody tools and extend perception using standard grasped objects. We propose that vibro-tactile sensing using dynamic tactile sensors on the robot fingers, along with machine learning models, enables robots to decipher contact information that is transmitted as vibrations along rigid objects.…
Effective human-robot collaboration requires informed anticipation. The robot must anticipate the human's actions, but also react quickly and intuitively when its predictions are wrong. The robot must plan its actions to account for the human's own plan, with the knowledge that the human's behavior will change based on what the robot actually does. This cyclical game of predicting a human's future actions and generating a corresponding motion plan is extremely difficult to model using standard techniques. In this work, we describe a novel Model Predictive Control (MPC)-based framework for find…
Towards Tangible Immersion for Cobot Programming-by-Demonstration: Visual, Tactile and Haptic Interfaces for Mixed-Reality Cobot Automation in Semiconductor Manufacturing
Intel Labs, Intelligent System Research, Human-Robot Collaboration（英特尔实验室、智能系统研究、人机协作）
Intent-Handover: Grounding Language in Human-Usage Regions for Trustworthy Robot-to-Human Handovers
DANiLab, University of Leicester（莱斯特大学DANiLab）；University of Leicester（莱斯特大学）；School of Computing and Mathematical Sciences, University of Leicester（莱斯特大学计算与数学科学学院）；School of Metallurgy and Materials, University of Birmingham（伯明翰大学冶金与材料学院）
TU Wien（维也纳工业大学）；Australian Center of Robotic Vision（澳大利亚机器人视觉中心）；Monash University（蒙纳士大学）；University of Birmingham（伯明翰大学）；Queensland University of Technology（昆士兰科技大学）
We present an approach for safe and object-independent human-to-robot handovers using real time robotic vision and manipulation. We aim for general applicability with a generic object detector, a fast grasp selection algorithm and by using a single gripper-mounted RGB-D camera, hence not relying on external sensors. The robot is controlled via visual servoing towards the object of interest. Putting a high emphasis on safety, we use two perception modules: human body part segmentation and hand/finger segmentation. Pixels that are deemed to belong to the human are filtered out from candidate gra…
Existing approaches to direct robot-to-human handovers are typically implemented on fixed-base robot arms, or on mobile manipulators that come to a full stop before performing the handover. We propose "on-the-go" handovers which permit a moving mobile manipulator to hand over an object to a human without stopping. The on-the-go handover motion is generated with a reactive controller that allows simultaneous control of the base and the arm. In a user study, human receivers subjectively assessed on-the-go handovers to be more efficient, predictable, natural, better timed and safer than handovers…
6D robotic grasping beyond top-down bin-picking scenarios is a challenging task. Previous solutions based on 6D grasp synthesis with robot motion planning usually operate in an open-loop setting, which are sensitive to grasp synthesis errors. In this work, we propose a new method for learning closed-loop control policies for 6D grasping. Our policy takes a segmented point cloud of an object from an egocentric camera as input, and outputs continuous 6D control actions of the robot gripper for grasping the object. We combine imitation learning and reinforcement learning and introduce a goal-auxi…
Although, in the task of grasping via a data-driven method, closed-loop feedback and predicting 6 degrees of freedom (DoF) grasp rather than conventionally used 4DoF top-down grasp are demonstrated to improve performance individually, few systems have both. Moreover, the sequential property of that task is hardly dealt with, while the approaching motion necessarily generates a series of observations. Therefore, this paper synthesizes three approaches and suggests a closed-loop framework that can predict the 6DoF grasp in a heavily cluttered environment from continuously received vision observa…
University of Stuttgart（斯图加特大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；TU Berlin（柏林工业大学）
Robotic manipulation of unknown objects is an important field of research. Practical applications occur in many real-world settings where robots need to interact with an unknown environment. We tackle the problem of reactive grasping by proposing a method for unknown object tracking, grasp point sampling and dynamic trajectory planning. Our object tracking method combines Siamese Networks with an Iterative Closest Point approach for pointcloud registration into a method for 6-DoF unknown object tracking. The method does not require further training and is robust to noise and occlusion. We prop…
Mitigating Compensatory Movements in Prosthesis Users via Adaptive Collaborative Robotics
Human-Robot Interfaces and Interaction, Istituto Italiano di Tecnologia（人机交互与交互，意大利技术研究院）；Dept. of Informatics, Bioengineering, Robotics and System Engineering, University of Genoa（信息学、生物工程、机器人与系统工程系，热那亚大学）
Stereo Hand-Object Reconstruction for Human-to-Robot Handover
Centre for Intelligent Sensing, Queen Mary University of London, UK（智能感知中心，伦敦女王玛丽大学）；Idiap Research Institute and École Polytechnique Fédérale de Lausanne, Switzerland（日内瓦联邦理工学院与IDiap研究机构）
Humans are highly skilled in communicating their intent for when and where a handover would occur. However, even the state-of-the-art robotic implementations for handovers typically lack of such communication skills. This study investigates visualization of the robot's internal state and intent for Human-to-Robot Handovers using Augmented Reality. Specifically, we explore the use of visualized 3D models of the object and the robotic gripper to communicate the robot's estimation of where the object is and the pose in which the robot intends to grasp the object. We tested this design via a user …
Heriot-Watt University（赫瑞-瓦特大学）；University of Washington（华盛顿大学）
Reasoning about object handover configurations allows an assistive agent to estimate the appropriateness of handover for a receiver with different arm mobility capacities. While there are existing approaches for estimating the effectiveness of handovers, their findings are limited to users without arm mobility impairments and to specific objects. Therefore, current state-of-the-art approaches are unable to hand over novel objects to receivers with different arm mobility capacities. We propose a method that generalises handover behaviours to previously unseen objects, subject to the constraint …
Robotic Surgical Assistants (RSAs) are commonly used to perform minimally invasive surgeries by expert surgeons. However, long procedures filled with tedious and repetitive tasks such as suturing can lead to surgeon fatigue, motivating the automation of suturing. As visual tracking of a thin reflective needle is extremely challenging, prior work has modified the needle with nonreflective contrasting paint. As a step towards automation of a suturing subtask without modifying the needle, we propose HOUSTON: Handoff of Unmodified, Surgical, Tool-Obstructed Needles, a problem and algorithm that us…
Toward Human-Robot Teaming: Learning Handover Behaviors from 3D Scenes
Centre for Intelligent Sensing, Queen Mary University of London（智能感知中心，伦敦女王玛丽大学）；Idiap Research Institute and École Polytechnique Fédérale de Lausanne（IDiap研究机构和日内瓦联邦理工学院）
NVIDIA（英伟达）；University of Toronto（多伦多大学）；University of Washington（华盛顿大学）
The pipeline of current robotic pick-and-place methods typically consists of several stages: grasp pose detection, finding inverse kinematic solutions for the detected poses, planning a collision-free trajectory, and then executing the open-loop trajectory to the grasp pose with a low-level tracking controller. While these grasping methods have shown good performance on grasping static objects on a table-top, the problem of grasping dynamic objects in constrained environments remains an open problem. We present Neural Motion Fields, a novel object representation which encodes both object point…
University of Bristol（布里斯托大学）；University of Pisa（比萨大学）；Istituto Italiano di Tecnologia（意大利技术研究院）
This paper presents a control scheme for force sensitive, gentle grasping with a Pisa/IIT anthropomorphic SoftHand equipped with a miniaturised version of the TacTip optical tactile sensor on all five fingertips. The tactile sensors provide high-resolution information about a grasp and how the fingers interact with held objects. We first describe a series of hardware developments for performing asynchronous sensor data acquisition and processing, resulting in a fast control loop sufficient for real-time grasp control. We then develop a novel grasp controller that uses tactile feedback from all…
