Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. Conventional methods directly consider the sparse human instruction to generate action plans for agents, which usually fail to achieve human goals because of the instruction incoherence in action descriptions. On the contrary, we propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions, so that the agent can successfully complete human goals by following the coherent instruction. Specifically, we first design an instruction completer based on large language models to recover the missing actions with interacted objects between consecutive human instruction, where the perceived surrounding environments and the completed sub-goals are considered for instruction completion. Based on the partially observed scene semantic maps, we present an object localizer to infer the position of interacted objects for agents to achieve complex human goals. Extensive experiments in the simulated environment show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.
Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Model
University of Chinese Academy of Sciences (UCAS)(中国科学院大学);Tsinghua University(清华大学);JD Explore Academy(京东探索学院);ShanghaiTech University(上海科技大学);Nanjing University(南京大学)
EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks
Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);University of Technology Sydney(悉尼科技大学)
Guide, Think, Act: Interactive Embodied Reasoning in Vision-Language-Action Models
Futian Laboratory(福田实验室);Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);International Digital Economy Academy (IDEA)(国际数字经济学院(IDEA));School of Robotics, Hunan University(湖南大学机器人学院);South China University of Technology(华南理工大学);Visincept(Visincept公司);National Key Laboratory of Smart Farm Technologies and Systems(智能农业技术与系统国家重点实验室)
AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
University of Science and Technology of China(科学技术大学);Singapore University of Technology and Design(新加坡科技设计大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);DAMO Academy, Alibaba Group(阿里巴巴集团大模型学院);Nanjing Institute of Software Technology(南京软件技术研究所);Nanjing University of Posts and Telecommunications(南京邮电大学);Hohai University(河海大学)
VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);University of Oxford(牛津大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(教育部下一代智能搜索与推荐工程研究中心);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)
ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representations
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Meituan Inc.(美团公司);Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Research Center for Space Computing System, Zhejiang Lab(浙江实验室空间计算系统研究中心);College of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院);School of Robotics and Automation, Nanjing University(南京大学机器人与自动化学院);College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院)
Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought
Northwestern University(西北大学);Northeastern University(东北大学);South China University of Technology(华南理工大学);Hong Kong Baptist University(香港 Baptist大学);Beijing Normal - Hong Kong Baptist University(北京师范大学-香港 Baptist大学)
NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning
Sun Yat-sen University(中山大学);Shanghai Jiao Tong University(上海交通大学);Peng Cheng Laboratory(鹏城实验室);University of Hong Kong(香港大学);Dataa Robotics(达闼机器人公司);University of Science and Technology of China(中国科学技术大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models
School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学);Research Center for SCIR, Harbin Institute of Technology(SCIR研究中心,哈尔滨工业大学);NExT Research Center, National University of Singapore(NExT研究中心,新加坡国立大学)
EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoning
Shanghai Jiao Tong University(上海交通大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Peng Cheng Laboratory(鹏城实验室);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Mohamed Bin Zayed University of Artificial Intelligence(Mohamed Bin Zayed人工智能大学);Yinwang Intelligent Technology Co.(亿维智能技术有限公司)
PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments
Beijing Jiaotong University(北京交通大学);Tencent Robotics X & Futian Laboratory, Shenzhen(腾讯机器人X及福田实验室,深圳);Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(基础模型研究中心,中国科学院自动化研究所);ObjectEye Inc(ObjectEye公司)
Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model
The Hong Kong University of Science and Technology (GZ)(香港科技大学(广州));National University of Singapore(新加坡国立大学);ShanghaiTech University(上海科技大学);East China Normal University(华东师范大学);Nanjing University of Information Science & Technology(南京信息工程大学);Zhejiang University(浙江大学);Institute of Automation, Chinese Academy of Science(中国科学院自动化研究所);Shanghai AI Laboratory(上海人工智能实验室)
VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies
Zhejiang University(浙江大学);Cornell University(康奈尔大学);National University of Singapore(新加坡国立大学);Xi'an University of Electronic Science and Technology(西安电子科技大学)