As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual - affecting RGB inputs - and dynamics - affecting transition dynamics - corruptions. Most recent efforts in visual navigation have typically focused on generalizing to novel target environments with similar appearance and dynamics characteristics. With RobustNav, we find that some standard embodied navigation agents significantly underperform (or fail) in the presence of visual or dynamics corruptions. We systematically analyze the kind of idiosyncrasies that emerge in the behavior of such agents when operating under corruptions. Finally, for visual corruptions in RobustNav, we show that while standard techniques to improve robustness such as data-augmentation and self-supervised adaptation offer some zero-shot resistance and improvements in navigation performance, there is still a long way to go in terms of recovering lost performance relative to clean "non-corrupt" settings, warranting more research in this direction. Our code is available at https://github.com/allenai/robustnav
Robots excel in performing repetitive and precision-sensitive tasks in controlled environments such as warehouses and factories, but have not been yet extended to embodied AI agents providing assistance in household tasks. Inspired by the catalyzing effect that benchmarks have played in the AI fields such as computer vision and natural language processing, the community is looking for new benchmarks for embodied AI. Prior work in embodied AI benchmark defines tasks using a different formalism, often specific to one environment, simulator or domain, making it hard to develop general and compara…
RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation
School of Cyber Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；National University of Singapore（新加坡国立大学）；Sun Yat-sen University（中山大学）；Beihang University（北京航空航天大学）
NavTrust: Benchmarking Trustworthiness for Embodied Navigation
Trustworthy Autonomous Systems Laboratory at the University of California, Riverside（加州大学河滨分校可信自主系统实验室）；University of Michigan（密歇根大学）；Workday；University of Southern California（南加州大学）；Texas A&M University（德克萨斯A&M大学）；Lehigh University（莱斯大学）
Multimodal Data Storage and Retrieval for Embodied AI: A Survey
School of Economics and Management, South China Normal University（经济管理学院，华南师范大学）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）
Australian National University（澳大利亚国立大学）；Princeton University（普林斯顿大学）；Carnegie Mellon University（卡内基梅隆大学）；Intel Labs（英特尔实验室）；UC Berkeley（加州大学伯克利分校）；George Mason University（乔治梅森大学）；Facebook（脸书）；Allen Institute for AI（艾伦人工智能研究所）
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, …
UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；State Key Laboratory of Complex & Critical Software Environment, Beihang University（北航复杂与关键软件环境国家重点实验室）；School of Computer Science, Institute for Artificial Intelligence, State Key Laboratory of General Artificial Intelligence, Peking University（北京大学计算机学院）；City University of Macau（澳门城市大学）；National University of Singapore（新加坡国立大学）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）
Georgia Tech（佐治亚理工学院）；Facebook AI Research（Facebook人工智能研究院）；Simon Fraser University（西蒙菲莎大学）；Imperial College London（帝国理工学院）；Princeton University（普林斯顿大学）；Intel Labs（英特尔实验室）；UC Berkeley（加州大学伯克利分校）；Google（谷歌）；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；UC San Diego（加州大学圣地亚哥分校）
We describe a framework for research and evaluation in Embodied AI. Our proposal is based on a canonical task: Rearrangement. A standard task can focus the development of new techniques and serve as a source of trained models that can be transferred to other settings. In the rearrangement task, the goal is to bring a given physical environment into a specified state. The goal state can be specified by object poses, by images, by a description in language, or by letting the agent experience the environment in the goal state. We characterize rearrangement scenarios along different axes and descr…
Facebook AI Research（脸书人工智能研究院）；Simon Fraser University（西蒙弗雷泽大学）；Facebook Reality Labs（脸书现实实验室）；Georgia Institute of Technology（佐治亚理工学院）；Intel Labs（英特尔实验室）；UC Berkeley（加州大学伯克利分校）
We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation. Specifically, Habitat consists of: (i) Habitat-Sim: a flexible, high-performance 3D simulator with configurable agents, sensors, and generic 3D dataset handling. Habitat-Sim is fast -- when rendering a scene from Matterport3D, it achieves several thousand frames per second (fps) running single-threaded, and can reach over 10,000 fps multi-process on a single GPU. (ii) Habitat-API: a modular high-level l…
Octopus: Embodied Vision-Language Programmer from Environmental Feedback
S-Lab, Nanyang Technological University（新加坡南洋理工大学S-Lab）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；BUPT（北京邮电大学）；XJTU（西安交通大学）；Hong Kong Baptist University（香港浸会大学）
Navigation tasks in photorealistic 3D environments are challenging because they require perception and effective planning under partial observability. Recent work shows that map-like memory is useful for long-horizon navigation tasks. However, a focused investigation of the impact of maps on navigation tasks of varying complexity has not yet been performed. We propose the multiON task, which requires navigation to an episode-specific sequence of objects in a realistic environment. MultiON generalizes the ObjectGoal navigation task and explicitly tests the ability of navigation agents to locate…
KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding
Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）
WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform
Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Zhejiang University（浙江大学）；Stanford University（斯坦福大学）；The University of Hong Kong（香港大学）；Princeton University（普林斯顿大学）；Chinese Academy of Sciences（中国科学院）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；National University of Singapore（新加坡国立大学）
SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments
The University of New South Wales（新南威尔士大学）；The University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）；Xinjiang University（新疆大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Openfly: A comprehensive platform for aerial vision-language navigation
Shanghai AI Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Beihang University（北航）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；East China University of Science and Technology（东华大学）；Fudan University（复旦大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；TeleAI
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook人工智能研究院）；University of Pennsylvania（宾夕法尼亚大学）
Embodied computer vision considers perception for robots in novel, unstructured environments. Of particular importance is the embodied visual exploration problem: how might a robot equipped with a camera scope out a new environment? Despite the progress thus far, many basic questions pertinent to this problem remain unanswered: (i) What does it mean for an agent to explore its environment well? (ii) Which methods work well, and under which assumptions and environmental settings? (iii) Where do current approaches fall short, and where might future work seek to improve? Seeking answers to these …
Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation
Sapienza University of Rome（罗马大学）；University of Verona（维罗纳大学）；Polytechnic of Turin（都灵理工大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；Reykjavik University（雷克雅未克大学）
GenRL: Multimodal-foundation world models for generalization in embodied agents
IDLab, Ghent University（IDLab，根特大学）；VERSES AI Research Lab（VERSES AI研究实验室）；Mila, University of Montreal（米拉研究所，蒙特利尔大学）；ServiceNow Research（ServiceNow研究院）
Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；BNRist, Tsinghua University（清华大学BNRist）
MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；University of California, Los Angeles（加州大学洛杉矶分校）；University of California, Berkeley（加州大学伯克利分校）
PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；ModelBest Inc.（ModelBest公司）；State Key Lab. of Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Institute for AI, Tsinghua University（清华大学人工智能研究院）
ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representations
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；Meituan Inc.（美团公司）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Research Center for Space Computing System, Zhejiang Lab（浙江实验室空间计算系统研究中心）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；School of Robotics and Automation, Nanjing University（南京大学机器人与自动化学院）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）
