Can world knowledge learned by large language models (LLMs) be used to act in interactive environments? In this paper, we investigate the possibility of grounding high-level tasks, expressed in natural language (e.g. "make breakfast"), to a chosen set of actionable steps (e.g. "open fridge"). While prior work focused on learning from explicit step-by-step examples of how to act, we surprisingly find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into mid-level plans without any further training. However, the plans produced naively by LLMs often cannot map precisely to admissible actions. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions. Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline. The conducted human evaluation reveals a trade-off between executability and correctness but shows a promising sign towards extracting actionable knowledge from language models. Website at https://huangwl18.github.io/language-planner
Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following
Seoul National University（首尔大学）；Interdisciplinary Program in AI (IPAI), Seoul National University（首尔大学人工智能交叉项目）；AI Institute, Seoul National University（首尔大学人工智能研究院）
Reasoning in a complex and ambiguous environment is a key goal for Reinforcement Learning (RL) agents. While some sophisticated RL agents can successfully solve difficult tasks, they require a large amount of training data and often struggle to generalize to new unseen environments and new tasks. On the other hand, Large Scale Language Models (LSLMs) have exhibited strong reasoning ability and the ability to to adapt to new tasks through in-context learning. However, LSLMs do not inherently have the ability to interrogate or intervene on the environment. In this work, we investigate how to com…
Beihang University（北京航空航天大学）；University of Technology Sydney（悉尼科技大学）；Meituan Inc.（美团公司）
Vision-language navigation is the task of directing an embodied agent to navigate in 3D scenes with natural language instructions. For the agent, inferring the long-term navigation target from visual-linguistic clues is crucial for reliable path planning, which, however, has rarely been studied before in literature. In this article, we propose a Target-Driven Structured Transformer Planner (TD-STP) for long-horizon goal-guided and room layout-aware navigation. Specifically, we devise an Imaginary Scene Tokenization mechanism for explicit estimation of the long-term target (even located in unex…
NormAct: Benchmarking Embodied Agents' Proactive Compliance with Unspoken Social Norms
State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（通用人工智能国家重点实验室，北京通用人工智能研究院）；China Academy of Information and Communications Technology（中国信息通信研究院）；ShanghaiTech University（上海科技大学）
Universal Actions for Enhanced Embodied Foundation Models
AIR, Tsinghua University（清华大学AIR）；Sensetime Research（商汤科技研究院）；Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai AI Lab（上海人工智能实验室）；BAAI（北京智源人工智能研究院）
Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
Shanghai AI Lab（上海人工智能实验室）；Fudan（复旦大学）；SJTU（上海交通大学）；NJUST（南京理工大学）；THU（清华大学）；Harvard（哈佛大学）；ZJU（浙江大学）；NJU（南京大学）；USTC（中国科学技术大学）；Tongji（同济大学）；HKUST (GZ)（香港科技大学（广州））
LogicGuard: Improving Embodied LLM agents through Temporal Logic based Critics
Department of Mechanical Engineering, University of California at Santa Barbara（加州大学圣芭芭拉分校机械工程系）；Department of Electrical and Computer Engineering, Michigan State University（密歇根州立大学电气与计算机工程系）
ContextFlow: Hierarchical Task-State Alignment for Long-Horizon Embodied Agents
Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Qiuzhen College, Tsinghua University（清华大学启祯学院）；Beijing Institute of Mathematical Sciences and Applications（北京数学科学研究院）；Department of Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）数据科学与分析部门）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration
Tsinghua University（清华大学）；Northwestern Polytechnical University（西北工业大学）；Institute of Artificial Intelligence (TeleAI)（人工智能研究所）；China Telecom（中国电信）；Zhejiang University（浙江大学）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）
Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning
Stanford University（斯坦福大学）；Politecnico di Milano（米兰理工大学）；KTH Royal Institute of Technology（皇家理工学院）；Italian Institute of Artificial Intelligence (AI4I)（意大利人工智能研究所）；NVIDIA Research（英伟达研究院）
CLEA: Closed-Loop Embodied Agent for Enhancing Task Execution in Dynamic Environments
Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)（深圳未来智能网络研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Infused Synapse AI；Harbin Engineering University（哈尔滨工程大学）；Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学医学机器人研究所）；School of Science and Engineering (SSE), FNii-Shenzhen（深圳未来智能网络研究院科学与工程学院）
Bridging Values and Behavior: A Hierarchical Framework for Proactive Embodied Agents
State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；School of Computer Science, Peking University（北京大学计算机学院）；Tsinghua University（清华大学）；Nat’l Eng. Research Center of Visual Technology, Peking University（北京大学视觉技术国家工程研究中心）；Institute for AI, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, Peking University（通用人工智能国家重点实验室，北京大学）
Never-Ending Behavior-Cloning Agent for Robotic Manipulation
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（机器人国家重点实验室，沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Automation Science and Engineering, South China University of Technology（自动化科学与工程学院，华南理工大学）；Mohamed bin Zayed University of Artificial Intelligence（马斯达尔大学人工智能学院）
