Robots assisting us in environments such as factories or homes must learn to make use of objects as tools to perform tasks, for instance using a tray to carry objects. We consider the problem of learning commonsense knowledge of when a tool may be useful and how its use may be composed with other tools to accomplish a high-level task instructed by a human. Specifically, we introduce a novel neural model, termed TOOLTANGO, that first predicts the next tool to be used, and then uses this information to predict the next action. We show that this joint model can inform learning of a fine-grained policy enabling the robot to use a particular tool in sequence and adds a significant value in making the model more accurate. TOOLTANGO encodes the world state, comprising objects and symbolic relationships between them, using a graph neural network and is trained using demonstrations from human teachers instructing a virtual robot in a physics simulator. The model learns to attend over the scene using knowledge of the goal and the action history, finally decoding the symbolic action to execute. Crucially, we address generalization to unseen environments where some known tools are missing, but alternative unseen tools are present. We show that by augmenting the representation of the environment with pre-trained embeddings derived from a knowledge-base, the model can generalize effectively to novel environments. Experimental results show at least 48.8-58.1% absolute improvement over the baselines in predicting successful symbolic plans for a simulated mobile manipulator in novel environments with unseen objects. This work takes a step in the direction of enabling robots to rapidly synthesize robust plans for complex tasks, particularly in novel settings
Predicting Stable Configurations for Semantic Placement of Novel Objects
NVIDIA Corporation(英伟达公司);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);School of Computing, University of Utah(犹他大学计算学院)
DyPNIPP: Predicting Environment Dynamics for RL-based Robust Informative Path Planning
Robotics Institute, Carnegie Mellon University(卡内基梅隆大学机器人研究所);Georgia Institute of Technology(佐治亚理工学院);University of North Carolina at Charlotte(北卡罗来纳大学夏洛特分校)
MEXGEN: An Effective and Efficient Information Gain Approximation for Information Gathering Path Planning
School of Computer & Mathematical Sciences, The University of Adelaide(阿德莱德大学计算机与数学科学学院);STELaRLab, Lockheed Martin Australia(洛克希德·马丁澳大利亚公司STELaRLab实验室)