Detecting Features of Tools, Objects, and Actions from Effects in a Robot using Deep Learning
Waseda University（早稻田大学）；National Institute of Informatics（国立信息学研究所）；SOKENDAI (the Graduate University for Advanced Studies)（综合研究大学院大学）；National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）
Namiko Saito, Kitae Kim, Shingo Murata, Tetsuya Ogata, Shigeki Sugano
We propose a tool-use model that can detect the features of tools, target objects, and actions from the provided effects of object manipulation. We construct a model that enables robots to manipulate objects with tools, using infant learning as a concept. To realize this, we train sensory-motor data recorded during a tool-use task performed by a robot with deep learning. Experiments include four factors: (1) tools, (2) objects, (3) actions, and (4) effects, which the model considers simultaneously. For evaluation, the robot generates predicted images and motions given information of the effects of using unknown tools and objects. We confirm that the robot is capable of detecting features of tools, objects, and actions by learning the effects and executing the task.
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Robot vision is greatly affected by occlusions, which poses challenges to autonomous systems. The robot itself may hide targets of interest from the camera, while it moves within the field of view, leading to failures in task execution. For example, if a target of interest is partially occluded by the robot, detecting and grasping it correctly, becomes very challenging. To solve this problem, we propose a computationally lightweight method to determine the areas that the robot occludes. For this purpose, we use the Unified Robot Description Format (URDF) to generate a virtual depth image of th…
Cornell University（康奈尔大学）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
We consider the problem of detecting robotic grasps in an RGB-D view of a scene containing objects. In this work, we apply a deep learning approach to solve this problem, which avoids time-consuming hand-design of features. This presents two main challenges. First, we need to evaluate a huge number of candidate grasps. In order to make detection fast, as well as robust, we present a two-step cascaded structure with two deep networks, where the top detections from the first are re-evaluated by the second. The first network has fewer features, is faster to run, and can effectively prune out unli…
The motion of robots and objects in our world is often highly dependent upon contact. When contact is expected but does not occur or when contact is not expected but does occur, robot behavior diverges from plan, often disastrously. This paper describes an approach that uses simulation to detect possible such behavioral divergences on real robots. This approach, and others like it, could be applied to validation of robot behaviors, mechanism design, and even online planning. The particle trace approach samples robot modeling parameters, sensory readings, and state estimates to evaluate a robot…
Delft University of Technology（代尔夫特理工大学）；Prince Sultan University（苏丹王子大学）
This paper introduces a compound vision system that enables robots to localize people up to 15m away using a cheap camera. And, it proposes a robust navigation stack that combines Deep Reinforcement Learning (DRL) and a probabilistic localization method. To test the efficacy of these systems, we prototyped a low-cost mobile robot that we call Covy. Covy can be used for applications such as promoting social distancing during pandemics or estimating the density of a crowd. We evaluated Covy's performance through extensive sets of experiments both in simulated and realistic environments. Our resu…
Human-robot collaboration has gained a notable prominence in Industry 4.0, as the use of collaborative robots increases efficiency and productivity in the automation process. However, it is necessary to consider the use of mechanisms that increase security in these environments, as the literature reports that risk situations may exist in the context of human-robot collaboration. One of the strategies that can be adopted is the visual recognition of the collaboration environment using machine learning techniques, which can automatically identify what is happening in the scene and what may happe…
Time-Unified Diffusion Policy with Action Discrimination for Robotic Manipulation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University（西安交通大学人工智能与机器人研究院）；School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics（西南财经大学计算机与人工智能学院）
This study uses multisensory data (i.e., color and depth) to recognize human actions in the context of multimodal human-robot interaction. Here we employed the iCub robot to observe the predefined actions of the human partners by using four different tools on 20 objects. We show that the proposed multimodal ensemble learning leverages complementary characteristics of three color cameras and one depth sensor that improves, in most cases, recognition accuracy compared to the models trained with a single modality. The results indicate that the proposed models can be deployed on the iCub robot tha…
In gaze based Human-Robot Interaction (HRI), it is important to determine the human intention for further interaction. The gaze intention is often modelled as fixation. However, when looking at an object, it is not natural and it is difficult to maintain the gaze fixating on one point for a long time. Saccades may happen while a human is still focusing on the object. The prediction of human intention will be lost during saccades. In addition, while the human intention is on object, the gazes may be located outside of the object bounding box due to different noise sources, which would cause fal…
Multiple opportunities lie at the intersection of multi-robot systems and distributed ledger technologies (DLTs). In this work, we investigate the potential of new DLT solutions such as IOTA, for detecting anomalies and byzantine agents in multi-robot systems in a decentralized manner. Traditional blockchain approaches are not applicable to real-world networked and decentralized robotic systems where connectivity conditions are not ideal. To address this, we leverage recent advances in partition-tolerant and byzantine-tolerant collaborative decision-making processes with IOTA smart contracts. …
Karelics Oy（卡雷利茨公司）；School of Computing, University of Eastern Finland（东芬兰大学计算机学院）；School of Electrical Engineering, Aalto University（阿尔托大学电气工程学院）
Deep reinforcement learning has proven to be successful for learning tasks in simulated environments, but applying same techniques for robots in real-world domain is more challenging, as they require hours of training. To address this, transfer learning can be used to train the policy first in a simulated environment and then transfer it to physical agent. As the simulation never matches reality perfectly, the physics, visuals and action spaces by necessity differ between these environments to some degree. In this work, we study how general video games can be directly used instead of fine-tune…
Virginia Tech（弗吉尼亚理工大学）；Stanford University（斯坦福大学）；University of Toronto（多伦多大学）
Assistive robot arms enable people with disabilities to conduct everyday tasks on their own. These arms are dexterous and high-dimensional; however, the interfaces people must use to control their robots are low-dimensional. Consider teleoperating a 7-DoF robot arm with a 2-DoF joystick. The robot is helping you eat dinner, and currently you want to cut a piece of tofu. Today's robots assume a pre-defined mapping between joystick inputs and robot actions: in one mode the joystick controls the robot's motion in the x-y plane, in another mode the joystick controls the robot's z-yaw motion, and s…
Robots interacting with the physical world plan with models of physics. We advocate that robots interacting with people need to plan with models of cognition. This writeup summarizes the insights we have gained in integrating computational cognitive models of people into robotics planning and control. It starts from a general game-theoretic formulation of interaction, and analyzes how different approximations result in different useful coordination behaviors for the robot during its interaction with people.
Stanford University（斯坦福大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Nvidia（英伟达）
Assistive robotic arms enable users with physical disabilities to perform everyday tasks without relying on a caregiver. Unfortunately, the very dexterity that makes these arms useful also makes them challenging to teleoperate: the robot has more degrees-of-freedom than the human can directly coordinate with a handheld joystick. Our insight is that we can make assistive robots easier for humans to control by leveraging latent actions. Latent actions provide a low-dimensional embedding of high-dimensional robot behavior: for example, one latent dimension might guide the assistive arm along a po…
Waseda University（早稻田大学）；Fujitsu Laboratories LTD.（富士通研究所）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
We propose a promising neural network model with which to acquire a grounded representation of robot actions and the linguistic descriptions thereof. Properly responding to various linguistic expressions, including polysemous words, is an important ability for robots that interact with people via linguistic dialogue. Previous studies have shown that robots can use words that are not included in the action-description paired datasets by using pre-trained word embeddings. However, the word embeddings trained under the distributional hypothesis are not grounded, as they are derived purely from a …
Recent advances in robotic mobile manipulation have spurred the expansion of the operating environment for robots from constrained workspaces to large-scale, human environments. In order to effectively complete tasks in these spaces, robots must be able to perceive, reason, and execute over a diversity of affordances, well beyond simple pick-and-place. We posit the notion of semantic frames provides a compelling representation for robot actions that is amenable to action-focused perception, task-level reasoning, action-level execution, and integration with language. Semantic frames, a product …
CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations
Department of Computer Science, University of Southern California（南加州大学计算机科学系）；Department of Electrical and Computer Engineering, University of Southern California（南加州大学电气与计算机工程系）
We propose and demonstrate the task of giving natural language summaries of the actions of a robotic agent in a virtual environment. We explain why such a task is important, what makes it difficult, and discuss how it might be addressed. To encourage others to work on this, we show how a popular existing dataset that matches robot actions with natural language descriptions designed for an instruction following task can be repurposed to serve as a training ground for robot action summarization work. We propose and test several methods of learning to generate such summaries, starting from either…
Korea Advanced Institute of Science and Technology（韩国高等科学技术院）；Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）
In this study, we investigate how a robot can generate novel and creative actions from its own experience of learning basic actions. Inspired by a machine learning approach to computational creativity, we propose a dynamic neural network model that can learn and generate robot's actions. We conducted a set of simulation experiments with a humanoid robot. The results showed that the proposed model was able to learn the basic actions and also to generate novel actions by modulating and combining those learned actions. The analysis on the neural activities illustrated that the ability to generate…
Programming robots for general purpose applications is extremely challenging due to the great diversity of end-user tasks ranging from manufacturing environments to personal homes. Recent work has focused on enabling end-users to program robots using Programming by Demonstration. However, teaching robots new actions from scratch that can be reused for unseen tasks remains a difficult challenge and is generally left up to robotic experts. We propose iRoPro, an interactive Robot Programming framework that allows end-users to teach robots new actions from scratch and reuse them with a task planne…
Georgia Institute of Technology（佐治亚理工学院）；Monash University（莫纳什大学）
The increasing presence of robots alongside humans, such as in human-robot teams in manufacturing, gives rise to research questions about the kind of behaviors people prefer in their robot counterparts. We term actions that support interaction by reducing future interference with others as supportive robot actions and investigate their utility in a co-located manipulation scenario. We compare two robot modes in a shared table pick-and-place task: (1) Task-oriented: the robot only takes actions to further its own task objective and (2) Supportive: the robot sometimes prefers supportive actions …
Identifying Influential Actions in Human-Robot Interactions
Guardian Robot Project, R-IH, Riken, Kyoto, Japan（守卫机器人项目，R-IH，Riken，京都，日本）；Faculty of Engineering, Monash University, Melbourne, Australia（工程学院，墨尔本大学，澳大利亚）；Graduate school of engineering science, Osaka University, Osaka, Japan（工程科学研究生院，大阪大学，日本）；Faculty of engineering, Tokyo University of Science, Tokyo, Japan（工程学院，东京科学大学，日本）
Universität Göttingen（哥廷根大学）；Vytautas Magnus University（维陶塔斯·马格努斯大学）
Efficient action prediction is of central importance for the fluent workflow between humans and equally so for human-robot interaction. To achieve prediction, actions can be encoded by a series of events, where every event corresponds to a change in a (static or dynamic) relation between some of the objects in a scene. Manipulation actions and others can be uniquely encoded this way and only, on average, less than 60% of the time series has to pass until an action can be predicted. Using a virtual reality setup and testing ten different manipulation actions, here we show that in most cases hum…
Despite the potential of reinforcement learning (RL) for building general-purpose robotic systems, training RL agents to solve robotics tasks still remains challenging due to the difficulty of exploration in purely continuous action spaces. Addressing this problem is an active area of research with the majority of focus on improving RL methods via better optimization or more efficient exploration. An alternate but important component to consider improving is the interface of the RL algorithm with the robot. In this work, we manually specify a library of robot action primitives (RAPS), paramete…
In human-robot teams, humans often start with an inaccurate model of the robot capabilities. As they interact with the robot, they infer the robot's capabilities and partially adapt to the robot, i.e., they might change their actions based on the observed outcomes and the robot's actions, without replicating the robot's policy. We present a game-theoretic model of human partial adaptation to the robot, where the human responds to the robot's actions by maximizing a reward function that changes stochastically over time, capturing the evolution of their expectations of the robot's capabilities. …
