ToolNet: Using Commonsense Generalization for Predicting Tool Use for Robot Plan Synthesis
Indian Institute of Technology Delhi(德里印度理工学院)
Rajas Bansal, Shreshth Tuli, Rohan Paul, Mausam
备注:
Advances & Challenges in Imitation Learning for Robotics Workshop in Robotics Science and Systems (RSS) 2020. The first two authors contributed equally
A robot working in a physical environment (like home or factory) needs to learn to use various available tools for accomplishing different tasks, for instance, a mop for cleaning and a tray for carrying objects. The number of possible tools is large and it may not be feasible to demonstrate usage of each individual tool during training. Can a robot learn commonsense knowledge and adapt to novel settings where some known tools are missing, but alternative unseen tools are present? We present a neural model that predicts the best tool from the available objects for achieving a given declarative goal. This model is trained by user demonstrations, which we crowd-source through humans instructing a robot in a physics simulator. This dataset maintains user plans involving multi-step object interactions along with symbolic state changes. Our neural model, ToolNet, combines a graph neural network to encode the current environment state, and goal-conditioned spatial attention to predict the appropriate tool. We find that providing metric and semantic properties of objects, and pre-trained object embeddings derived from a commonsense knowledge repository such as ConceptNet, significantly improves the model's ability to generalize to unseen tools. The model makes accurate and generalizable tool predictions. When compared to a graph neural network baseline, it achieves 14-27% accuracy improvement for predicting known tools from new world scenes, and 44-67% improvement in generalization for novel objects not encountered during training.
PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification
Department of Computer Science and Operations Research, Université de Montréal(蒙特利尔大学计算机科学与运筹学系);Mila - Quebec AI Institute(魁北克人工智能研究所);CIFAR AI Chair(CIFAR人工智能主席)
Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning
Department of Computer, Automation and Management Engineering, Sapienza University of Rome(Sapienza大学计算机、自动化与管理工程系);Department of Electronics and Automation, Mohamed Khider University of Biskra(Mohamed Khider大学电子与自动化系);International University of Rome UNINT, Rome(罗马国际大学UNINT)
Multi-Robot Task Planning for Multi-Object Retrieval Tasks with Distributed On-Site Knowledge via Large Language Models
Graduate School of Information Science and Engineering(信息科学与工程研究生学校);Ritsumeikan University(立命馆大学);College of Information Science and Engineering(信息科学与工程学院);Faculty of Science and Engineering(科学与工程学部);Soka University(早稻田大学);Research Organization of Science and Technology(科学与技术研究机构);Graduate School of Informatics(信息学研究生学校);Kyoto University(京都大学)
OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on Dynamic Carrier-Relationship Scene Graph
School of Automation, Beijing Institute of Technology(北京理工大学自动化学院);School of Mechanical Engineering, University of Nottingham Ningbo China(宁波诺丁汉大学机械工程学院)