Embodied Question Answering (EQA) is a recently proposed task, where an agent is placed in a rich 3D environment and must act based solely on its egocentric input to answer a given question. The desired outcome is that the agent learns to combine capabilities such as scene understanding, navigation and language understanding in order to perform complex reasoning in the visual world. However, initial advancements combining standard vision and language methods with imitation and reinforcement learning algorithms have shown EQA might be too complex and challenging for these techniques. In order to investigate the feasibility of EQA-type tasks, we build the VideoNavQA dataset that contains pairs of questions and videos generated in the House3D environment. The goal of this dataset is to assess question-answering performance from nearly-ideal navigation paths, while considering a much more complete variety of questions than current instantiations of the EQA task. We investigate several models, adapted from popular VQA methods, on this new benchmark. This establishes an initial understanding of how well VQA-style methods can perform within this novel EQA paradigm.
We introduce ROS-X-Habitat, a software interface that bridges the AI Habitat platform for embodied learning-based agents with other robotics resources via ROS. This interface not only offers standardized communication protocols between embodied agents and simulators, but also enables physically and photorealistic simulation that benefits the training and/or testing of vision-based embodied agents. With this interface, roboticists can evaluate their own Habitat RL agents in another ROS-based simulator or use Habitat Sim v2 as the test bed for their own robotic algorithms. Through in silico expe…
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；King Abdullah University of Science and Technology（科廷大学）；Fudan University（复旦大学）
EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)（博世北美研究部及博世人工智能中心（BCAI））
Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
Northwestern Polytechnical University（西北工业大学）；The University of Georgia（佐治亚大学）；Shaanxi Normal University（陕西师范大学）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）
See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm
University College London（伦敦大学学院）；Imperial College London（伦敦帝国学院）；Huawei Noah’s Ark Lab（华为诺亚实验室）；ShanghaiTech University（上海科技大学）；Southern University of Science（南方科技大学）
Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Bosch Corporate Research（博世企业研究）；King Abdullah University of Science and Technology（卡布斯大学）
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Beihang University（北航）；State Key Laboratory of General Artificial Intelligence, BIGAI, China（国家一般人工智能重点实验室, BIGAI, 中国）
BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences（模式识别新实验室（NLPR），自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Shandong University（山东大学）
PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly
Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·扎耶德人工智能大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University, China（清华大学深圳国际研究生院）；Shenzhen Campus of Sun Yat-sen University（孙中山大学深圳校区）；Spatialtemporal AI（时空人工智能）；Shanghai Jiao Tong University（上海交通大学）
ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models
School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）；Research Center for SCIR, Harbin Institute of Technology（SCIR研究中心，哈尔滨工业大学）；NExT Research Center, National University of Singapore（NExT研究中心，新加坡国立大学）
Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Zhejiang University（浙江大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）；NUS（新加坡国立大学）；Northeastern University（东北大学）；Shenzhen University（深圳大学）
Despite the impressive progress brought by deep network in visual object recognition, robot vision is still far from being a solved problem. The most successful convolutional architectures are developed starting from ImageNet, a large scale collection of images of object categories downloaded from the Web. This kind of images is very different from the situated and embodied visual experience of robots deployed in unconstrained settings. To reduce the gap between these two visual experiences, this paper proposes a simple yet effective data augmentation layer that zooms on the object of interest…
PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations
Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）
EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents
The University of Hong Kong（香港大学）；Tampere University（坦佩雷大学）；The Chinese University of Hong Kong（香港中文大学）；Max-Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Neural Brain: A Neuroscience-inspired Framework for Embodied Agents
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）智能科学与工程学院）；Department of Information and Communication Technology, University of Agder（阿格德大学信息与通信技术系）；Department of Information Engineering and Computer Science, University of Trento（特伦特大学信息工程与计算机科学系）；Elm Company（Elm公司）；Division of Computational Science and Technology, KTH Royal Institute of Technology（皇家理工学院计算科学与技术系）；School of Artificial Intelligence and Data Science, Sejong University（世宗大学人工智能与数据科学学院）；Department of Computer Science of the University of Western Australia（西澳大学计算机科学系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments
AutoLab, SAI, Shanghai Jiao Tong University（上海交通大学SAI学院AutoLab实验室）；AIM3 Lab, School of Information, Renmin University of China（中国人民大学信息学院AIM3实验室）；Research Lab, Anyverse Dynamics（Anyverse Dynamics研究实验室）
Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inference
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；BAAI（北京人工智能研究院）
Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension
Institute of Deep Perception Technology, JITRI（JITRI深度感知技术研究所）；University of Liverpool（利物浦大学）；University of Southampton（南安普顿大学）；Momoni AI（Momoni人工智能公司）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Thrust of Artificial Intelligence, HKUST (GZ)（香港科技大学（广州）人工智能学域）；Thrust of Intelligent Transportation, HKUST (GZ)（香港科技大学（广州）智能交通学域）
Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos
Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence（计算机视觉系，Mohamed Bin Zayed人工智能大学）；School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）；Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）
State Key Laboratory of Precision Blasting, Jianghan University（江汉大学精密爆破重点实验室）；Harvard AI and Robotics Lab, Harvard University（哈佛大学AI与机器人实验室）；School of EIC, Huazhong University of Science and Technology（华中科技大学电子信息学院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机科学系）；School of Mathematics and Statistics, Hubei University of Education（湖北省教育学院数学与统计学学院）
