This paper investigates robot manipulation based on human instruction with ambiguous requests. The intent is to compensate for imperfect natural language via visual observations. Early symbolic methods, based on manually defined symbols, built modular framework consist of semantic parsing and task planning for producing sequences of actions from natural language requests. Modern connectionist methods employ deep neural networks to automatically learn visual and linguistic features and map to a sequence of low-level actions, in an endto-end fashion. These two approaches are blended to create a hybrid, modular framework: it formulates instruction following as symbolic goal learning via deep neural networks followed by task planning via symbolic planners. Connectionist and symbolic modules are bridged with Planning Domain Definition Language. The vision-and-language learning network predicts its goal representation, which is sent to a planner for producing a task-completing action sequence. For improving the flexibility of natural language, we further incorporate implicit human intents with explicit human instructions. To learn generic features for vision and language, we propose to separately pretrain vision and language encoders on scene graph parsing and semantic textual similarity tasks. Benchmarking evaluates the impacts of different components of, or options for, the vision-and-language learning model and shows the effectiveness of pretraining strategies. Manipulation experiments conducted in the simulator AI2THOR show the robustness of the framework to novel scenarios.
Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision
University of Padova, Dept. of Information Engineering（帕多瓦大学信息工程系）；Fraunhofer Italia Research（弗劳恩霍夫意大利研究所）；Polytechnic of Bari Dept. of Electrical and Information Engineering（巴里理工学院电气与信息工程系）
Lang2Manip: A Tool for LLM-Based Symbolic-to-Geometric Planning for Manipulation
Khalifa University Center for Autonomous Robotic Systems (KUCARS)（卡利法大学自主机器人系统中心）；Khalifa University（卡利法大学）；Institute of Industrial and Control Engineering (IOC)（工业与控制工程研究所）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
NEXUS: Learning Safety Constraints and Action Models for Safe and Robust Embodied Planning
School of Artificial Intelligence and Automation（人工智能与自动化学院）；Huazhong University of Science and Technology（华中科技大学）；School of Software（软件学院）；Henan University（河南大学）；Institute of Information Engineering（信息工程研究所）；Chinese Academy of Sciences（中国科学院）；School of Cyberspace Security（网络空间安全学院）；University of the Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）
LLM-Grounded Dynamic Task Planning with Hierarchical Temporal Logic for Human-Aware Multi-Robot Handover
The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人社会研究院）；Harbin Institute of Technology（哈尔滨工业大学）；The Chinese University of Hong Kong-Shenzhen（香港中文大学（深圳））；Tsinghua Shenzhen International Graduate School（清华大学深圳国际 Graduate School）
