Robots that must operate in novel environments and collaborate with humans must be capable of acquiring new knowledge from human experts during operation. We propose teaching a robot novel objects it has not encountered before by pointing a hand at the new object of interest. An end-to-end neural network is used to attend to the novel object of interest indicated by the pointing hand and then to localize the object in new scenes. In order to attend to the novel object indicated by the pointing hand, we propose a spatial attention modulation mechanism that learns to focus on the highlighted object while ignoring the other objects in the scene. We show that a robot arm can manipulate novel objects that are highlighted by pointing a hand at them. We also evaluate the performance of the proposed architecture on a synthetic dataset constructed using emojis and on a real-world dataset of common objects.
In this paper a neuro-robotics model capable of counting using gestures is introduced. The contribution of gestures to learning to count is tested with various model and training conditions. Two studies were presented in this article. In the first, we combine different modalities of the robot's neural network, in the second, a novel training procedure for it is proposed. The model is trained with pointing data from an iCub robot simulator. The behaviour of the model is in line with that of human children in terms of performance change depending on gesture production.
To better interact with users, a social robot should understand the users' behavior, infer the intention, and respond appropriately. Machine learning is one way of implementing robot intelligence. It provides the ability to automatically learn and improve from experience instead of explicitly telling the robot what to do. Social skills can also be learned through watching human-human interaction videos. However, human-human interaction datasets are relatively scarce to learn interactions that occur in various situations. Moreover, we aim to use service robots in the elderly-care domain; howeve…
Technical University of Munich（慕尼黑工业大学）；Madras Institute of Technology, Anna University（安娜大学马德拉斯理工学院）
A robot operating in unstructured environments must be able to discriminate between different grasping styles depending on the prospective manipulation task. Having a system that allows learning from remote non-expert demonstrations can very feasibly extend the cognitive skills of a robot for task-oriented grasping. We propose a novel two-step framework towards this aim. The first step involves grasp area estimation by segmentation. We receive grasp area demonstrations for a new task via interactive segmentation, and learn from these few demonstrations to estimate the required grasp area on an…
University of Alberta（阿尔伯塔大学）；Cairo University（开罗大学）；Facebook AI Research（Facebook人工智能研究院）
Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments, since the total number of objects and their variations can be intractable. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without provid…
University of Plymouth（普利茅斯大学）；University of Manchester（曼彻斯特大学）；AIST-AIRC（日本产业技术综合研究所人工智能研究中心）；Sheffield Hallam University（谢菲尔德哈勒姆大学）
In this paper, a novel neuro-robotics model capable of counting real items is introduced. The model allows us to investigate the interaction between embodiment and numerical cognition. This is composed of a deep neural network capable of image processing and sequential tasks performance, and a robotic platform providing the embodiment - the iCub humanoid robot. The network is trained using images from the robot's cameras and proprioceptive signals from its joints. The trained model is able to count a set of items and at the same time points to them. We investigate the influence of pointing on …
Collaborative robotics requires effective communication between a robot and a human partner. This work proposes a set of interpretive principles for how a robotic arm can use pointing actions to communicate task information to people by extending existing models from the related literature. These principles are evaluated through studies where English-speaking human subjects view animations of simulated robots instructing pick-and-place tasks. The evaluation distinguishes two classes of pointing actions that arise in pick-and-place tasks: referential pointing (identifying objects) and locating …
Learning from demonstration allows for rapid deployment of robot manipulators to a great many tasks, by relying on a person showing the robot what to do rather than programming it. While this approach provides many opportunities, measuring, evaluating and improving the person's teaching ability has remained largely unexplored in robot manipulation research. To this end, a model for learning from demonstration is presented here which incorporates the teacher's understanding of, and influence on, the learner. The proposed model is used to clarify the teacher's objectives during learning from dem…
Tsinghua University（清华大学）；University of Bath（巴斯大学）
Pointing gestures are widely used in robot navigationapproaches nowadays. However, most approaches only use point-ing gestures, and these have two major limitations. Firstly, they need to recognize pointing gestures all the time, which leads to long processing time and significant system overheads. Secondly,the user's pointing direction may not be very accurate, so the robot may go to an undesired place. To relieve these limitations,we propose a voice-guided pointing robot navigation approach named VGPN, and implement its prototype on a wheeled robot,TurtleBot 2. VGPN recognizes a pointing ges…
School of Computing（计算学院）；Smart Systems Institute（智能系统研究所）；National University of Singapore（新加坡国立大学）
Knowledge and skills can transfer from human teachers to human students. However, such direct transfer is often not scalable for physical tasks, as they require one-to-one interaction, and human teachers are not available in sufficient numbers. Machine learning enables robots to become experts and play the role of teachers to help in this situation. In this work, we formalize cooperative robot teaching as a Markov game, consisting of four key elements: the target task, the student model, the teacher model, and the interactive teaching-learning process. Under a moderate assumption, the Markov g…
Classical control techniques such as PID and LQR have been used effectively in maintaining a system state, but these techniques become more difficult to implement when the model dynamics increase in complexity and sensitivity. For adaptive robotic locomotion tasks with several degrees of freedom, this task becomes infeasible with classical control techniques. Instead, reinforcement learning can train optimal walking policies with ease. We apply deep Q-learning and augmented random search (ARS) to teach a simulated two-dimensional bipedal robot how to walk using the OpenAI Gym BipedalWalker-v3 …
The use of robots in educational settings is growing increasingly popular. Yet, many of the learning tasks involving social robots do not take full advantage of their physical embodiment. MobiAxis is a proposed learning task which uses the physical capabilities of a Pepper robot to teach the concepts of positive and negative multiplication along a number line. The robot is embodied with a number of multi-modal socially intelligent features and behaviours which are designed to enhance learning. This paper is a position paper describing the technical and theoretical implementation of the task, a…
University of Hamburg（汉堡大学）；University of Amsterdam（阿姆斯特丹大学）
Whenever we are addressing a specific object or refer to a certain spatial location, we are using referential or deictic gestures usually accompanied by some verbal description. Especially pointing gestures are necessary to dissolve ambiguities in a scene and they are of crucial importance when verbal communication may fail due to environmental conditions or when two persons simply do not speak the same language. With the currently increasing advances of humanoid robots and their future integration in domestic domains, the development of gesture interfaces complementing human-robot interaction…
Real-time Human Finger Pointing Recognition and Estimation for Robot Directives Using a Single Web-Camera
Department of Mechanical Engineering, Massachusetts Institute of Technology (MIT)（麻省理工学院机械工程系）；School of Mechanical Engineering, Tel-Aviv University（特拉维夫大学机械工程学院）
Examining the legibility of humanoid robot arm movements in a pointing task
Faculty of Mathematics, Physics；Informatics, Comenius University Bratislava, Slovakia DIBRIS, University of Genoa, Genoa, Italy CONTACT, Italian Institute of Technology, Genoa, Italy Digital Health Dept., Nvision Systems \& Technologies, S.L., Barcelona, Spain Faculty of Education, University of Ljubljana, Slovenia
A household robot is expected to perform various manipulative operations with an understanding of the purpose of the task. To this end, a desirable robotic application should provide an on-site robot teaching framework for non-experts. Here we propose a Learning-from-Observation (LfO) framework for grasp-manipulation-release class household operations (GMR-operations). The framework maps human demonstrations to predefined task models through one-shot teaching. Each task model contains both high-level knowledge regarding the geometric constraints and low-level knowledge related to human posture…
National Institute of Advanced Industrial Science and Engineering（国立先进产业科学技术研究院）；Beihang University（北京航空航天大学）
The motivation of this paper is to develop a smart system using multi-modal vision for next-generation mechanical assembly. It includes two phases where in the first phase human beings teach the assembly structure to a robot and in the second phase the robot finds objects and grasps and assembles them using AI planning. The crucial part of the system is the precision of 3D visual detection and the paper presents multi-modal approaches to meet the requirements: AR markers are used in the teaching phase since human beings can actively control the process. Point cloud matching and geometric const…
Computer Science Department, RWTH Aachen University（鲁尔大学亚琛分校计算机科学系）；Cyber Physical Systems, Indian Institute of Science Bengaluru（印度科学研究院班加罗尔分校）；Alma Mater Studiorum Università di Bologna（博洛尼亚大学）；Cognitive Robotics Department, Delft University of Technology（代尔夫特理工大学认知机器人学系）
University of Applied Sciences Bielefeld（比勒费尔德应用科学大学）；Otto-von-Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）
The increasing number of robots in home environments leads to an emerging coexistence between humans and robots. Robots undertake common tasks and support the residents in their everyday life. People appreciate the presence of robots in their environment as long as they keep the control over them. One important aspect is the control of a robot's workspace. Therefore, we introduce virtual borders to precisely and flexibly define the workspace of mobile robots. First, we propose a novel framework that allows a person to interactively restrict a mobile robot's workspace. To show the validity of t…
This letter presents a physical human-robot interaction scenario in which a robot guides and performs the role of a teacher within a defined dance training framework. A combined cognitive and physical feedback of performance is proposed for assisting the skill learning process. Direct contact cooperation has been designed through an adaptive impedance-based controller that adjusts according to the partner's performance in the task. In measuring performance, a scoring system has been designed using the concept of progressive teaching (PT). The system adjusts the difficulty based on the user's n…
Given the laborious difficulty of moving heavy bags of physical currency in the cash center of the bank, there is a large demand for training and deploying safe autonomous systems capable of conducting such tasks in a collaborative workspace. However, the deformable properties of the bag along with the large quantity of rigid-body coins contained within it, significantly increases the challenges of bag detection, grasping and manipulation by a robotic gripper and arm. In this paper, we apply deep reinforcement learning and machine learning techniques to the task of controlling a collaborative …
A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo-Dex, a framework for dexterous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-pur…
Many high precision (dis)assembly tasks are still being performed by humans, whereas this is an ideal opportunity for automation. This paper provides a framework which enables a non-expert human operator to teach a robotic arm to do complex precision tasks. The framework uses a variable Cartesian impedance controller to execute trajectories learned from kinesthetic human demonstrations. Feedback can be given to interactively reshape or speed up the original demonstration. Board localization is done through a visual estimation of the task board position and refined through haptic feedback. Our …
University of Pennsylvania（宾夕法尼亚大学）；Technical University of Munich（慕尼黑工业大学）；Lehigh University（理海大学）；University of Texas Austin（德克萨斯大学奥斯汀分校）；Poznan University of Technology（波兹南理工大学）；University of Virginia（弗吉尼亚大学）；Clemson University（克莱姆森大学）；Toyota Research Institute（丰田研究院）
Although robotics courses are well established in higher education, the courses often focus on theory and sometimes lack the systematic coverage of the techniques involved in developing, deploying, and applying software to real hardware. Additionally, most hardware platforms for robotics teaching are low-level toys aimed at younger students at middle-school levels. To address this gap, an autonomous vehicle hardware platform, called F1TENTH, is developed for teaching autonomous systems hands-on. This article describes the teaching modules and software stack for teaching at various educational …
University of Tehran（德黑兰大学）；Intelligent Systems Research Institute, SKKU（成均馆大学智能系统研究所）；Allameh Tabatabái University（阿拉梅塔巴塔巴伊大学）；Iran University of Medical Sciences（伊朗医科大学）；Shahid Beheshti University（沙希德贝赫什提大学）
Robot Assisted Therapy is a new paradigm in many therapies such as the therapy of children with autism spectrum disorder. In this paper we present the use of a parrot-like robot as an assistive tool in turn taking therapy. The therapy is designed in the form of a card game between a child with autism and a therapist or the robot. The intervention was implemented in a single subject study format and the effect sizes for different turn taking variables are calculated. The results show that the child robot interaction had larger effect size than the child trainer effect size in most of the turn t…
Towards Robotic Laboratory Automation Plug & Play: Survey and Concept Proposal on Teaching-free Robot Integration with the LAPP Digital Twin
Takeda Manufacturing Austria AG（武田奥地利制造公司）；Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University（欧布达大学应用信息学与应用数学博士学院）；Óbuda University（欧布达大学）；Baxalta Innovations GmbH, a Takeda company（百特创新有限公司（武田旗下公司））；Alba Regia Technical Faculty, Óbuda University（欧布达大学阿尔巴雷吉亚技术学院）；Antal Bejczy Center for Intelligent Robotics, Óbuda University（欧布达大学安塔尔·贝伊奇智能机器人中心）
