Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions
Department of Automation, Tsinghua University(清华大学自动化系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息学院);Computer Science and Engineering Department, Chinese University of Hong Kong(香港中文大学计算机科学与工程系)
Enabling robots to explore and act in unfamiliar environments under ambiguous human instructions by interactively identifying task-relevant objects (e.g., cups or beverages for "I'm thirsty") remains challenging for existing vision-language model (VLM)-based methods, due to inefficient reasoning and limited environmental interaction. These factors hinder real-time task planning and execution. To address this, we propose Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions (AIDE), a human-inspired dual-stream framework that integrates interactive exploration with vision-language reasoning: mirroring the complementary deliberative and intuitive processes of human cognition, Multi-Stage Inference (MSI) serves as the decision-making stream and Accelerated Decision-Making (ADM) as the execution stream, enabling zero-shot affordance analysis and interpretation of ambiguous instructions. Experiments in simulation and real-world settings show that AIDE achieves over 80% task planning success rate and more than 95% closed-loop continuous execution accuracy at 10 Hz, surpassing existing VLM-based methods in diverse open scenarios.
Interactive Robot Learning of Gestures, Language and Affordances
Institute for Systems and Robotics(系统与机器人研究所);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等理工学院);Queen Mary University of London(伦敦玛丽女王大学);KTH Royal Institute of Technology(KTH皇家理工学院)
LangGrasp: Leveraging Fine-Tuned LLMs for Language Interactive Robot Grasping with Ambiguous Instructions
School of Computer Science and Technology(计算机科学与技术学院);Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System(湖北省智能信息处理与实时工业系统重点实验室);Institute of Robotics and Intelligent Systems(机器人与智能系统研究院);Wuhan University of Science and Technology(武汉理工大学)
Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation
Department of Computer Science, Rutgers University-New Brunswick(罗格斯大学新布朗斯维尔回声分校计算机科学系);The Hong Kong University of Science and Technology (GZ)(香港科学与技术大学(GZ));Shanghai AI Laboratory(上海人工智能实验室)
SAIN: Structure-Aware Interactive Navigation with Active Dialogue Grounding for Mobile Robot
School of Mechanical Engineering and Automation, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)机械工程与自动化学院);Department of Mechanical Engineering, City University of Hong Kong(香港城市大学机械工程系)
CoINS: Counterfactual Interactive Navigation via Skill-Aware VLM
School of Advanced Manufacturing and Robotics, Peking University(北京大学先进制造与机器人学院);Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong(香港中文大学机械与自动化工程系);College of Design and Engineering, National University Of Singapore(新加坡国立大学设计与工程学院);Department of Computer Science, The University of Manchester(曼彻斯特大学计算机科学系)