Vision-Language Models (VLMs) have shown remarkable progress in Vision-Language Navigation (VLN), offering new possibilities for navigation decision-making that could benefit both robotic platforms and human users. However, real-world navigation is inherently conditioned by the agent's mobility constraints. For example, a sweeping robot cannot traverse stairs, while a quadruped can. We introduce Capability-Conditioned Navigation (CapNav), a benchmark designed to evaluate how well VLMs can navigate complex indoor spaces given an agent's specific physical and operational capabilities. CapNav defines five representative human and robot agents, each described with physical dimensions, mobility capabilities, and environmental interaction abilities. CapNav provides 45 real-world indoor scenes, 473 navigation tasks, and 2365 QA pairs to test if VLMs can traverse indoor environments based on agent capabilities. We evaluate 13 modern VLMs and find that current VLM's navigation performance drops sharply as mobility constraints tighten, and that even state-of-the-art models struggle with obstacle types that require reasoning on spatial dimensions. We conclude by discussing the implications for capability-aware navigation and the opportunities for advancing embodied spatial reasoning in future VLMs. The benchmark is available at https://github.com/makeabilitylab/CapNav
Datasets have gained an enormous amount of popularity in the computer vision community, from training and evaluation of Deep Learning-based methods to benchmarking Simultaneous Localization and Mapping (SLAM). Without a doubt, synthetic imagery bears a vast potential due to scalability in terms of amounts of data obtainable without tedious manual ground truth annotations or measurements. Here, we present a dataset with the aim of providing a higher degree of photo-realism, larger scale, more variability as well as serving a wider range of purposes compared to existing datasets. Our dataset lev…
Lidar technology has evolved significantly over the last decade, with higher resolution, better accuracy, and lower cost devices available today. In addition, new scanning modalities and novel sensor technologies have emerged in recent years. Public datasets have enabled benchmarking of algorithms and have set standards for the cutting edge technology. However, existing datasets are not representative of the technological landscape, with only a reduced number of lidars available. This inherently limits the development and comparison of general-purpose algorithms in the evolving landscape. This…
Visual-inertial SLAM is essential for robot navigation in GPS-denied environments, e.g. indoor, underground. Conventionally, the performance of visual-inertial SLAM is evaluated with open-loop analysis, with a focus on the drift level of SLAM systems. In this paper, we raise the question on the importance of visual estimation latency in closed-loop navigation tasks, such as accurate trajectory tracking. To understand the impact of both drift and latency on visual-inertial SLAM systems, a closed-loop benchmarking simulation is conducted, where a robot is commanded to follow a desired trajectory…
Task-driven SLAM Benchmarking For Robot Navigation
School of Electrical and Computer Engineering, and Institute of Robotics and Intelligent Machines, Georgia Institute of Technology（电气与计算机工程学院，机器人与智能机器研究所，佐治亚理工学院）；School of Mechanical Engineering and the School of Electrical and Computer Engineering, Georgia Institute of Technology（机械工程学院，电气与计算机工程学院，佐治亚理工学院）
SMapper: A Multi-Modal Data Acquisition Platform for SLAM Benchmarking
Automation and Robotics Research Group (ARG), Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg（卢森堡大学自动化与机器人研究组（ARG）、安全、可靠与信任跨学科研究中心（SnT））
CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments
College of Systems Engineering, National University of Defense Technology（系统工程学院，国防科技大学）；State Key Laboratory of Digital Intelligent Modeling and Simulation（数字智能建模与仿真国家重点实验室）；BNRist, Tsinghua University（清华大学北京研究院）；Department of Electronic Engineering, Tsinghua University（电子工程系，清华大学）
PALoc: Advancing SLAM Benchmarking with Prior-Assisted 6-DoF Trajectory Generation and Uncertainty Estimation
Hong Kong University of Science and Technology（香港科技大学）；Hong Kong University of Science and Technology, Guangzhou Campus（香港科技大学广州校区）；South China Normal University（华南师范大学）；University College London（伦敦大学学院）
SpaceDG: Benchmarking Spatial Intelligence under Visual Degradation
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Electronic Science and Technology of China（电子科技大学）；Chongqing University（重庆大学）；The University of Tokyo（东京大学）；Beihang University（北航）；Northwestern Polytechnical University（西北工业大学）
Learning agents can optimize standard autonomous navigation improving flexibility, efficiency, and computational cost of the system by adopting a wide variety of approaches. This work introduces the \textit{PIC4rl-gym}, a fundamental modular framework to enhance navigation and learning research by mixing ROS2 and Gazebo, the standard tools of the robotics community, with Deep Reinforcement Learning (DRL). The paper describes the whole structure of the PIC4rl-gym, which fully integrates DRL agent's training and testing in several indoor and outdoor navigation scenarios and tasks. A modular appr…
We present a novel dataset for training and benchmarking semantic SLAM methods. The dataset consists of 200 long sequences, each one containing 3000-5000 data frames. We generate the sequences using realistic home layouts. For that we sample trajectories that simulate motions of a simple home robot, and then render the frames along the trajectories. Each data frame contains a) RGB images generated using physically-based rendering, b) simulated depth measurements, c) simulated IMU readings and d) ground truth occupancy grid of a house. Our dataset serves a wider range of purposes compared to ex…
MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；University of California, Los Angeles（加州大学洛杉矶分校）；University of California, Berkeley（加州大学伯克利分校）
Navigation research is attracting renewed interest with the advent of learning-based methods. However, this new line of work is largely disconnected from well-established classic navigation approaches. In this paper, we take a step towards coordinating these two directions of research. We set up classic and learning-based navigation systems in common simulated environments and thoroughly evaluate them in indoor spaces of varying complexity, with access to different sensory modalities. Additionally, we measure human performance in the same environments. We find that a classic pipeline, when pro…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；George Mason University（乔治梅森大学）
We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the da…
Openfly: A comprehensive platform for aerial vision-language navigation
Shanghai AI Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Beihang University（北航）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；East China University of Science and Technology（东华大学）；Fudan University（复旦大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；TeleAI
RVN-Bench: A Benchmark for Reactive Visual Navigation
Department of Electrical and Computer Engineering, Seoul National University (SNU) and Automation and Systems Research Institute (ASRI) and Sequor Robotics Inc.（电气与计算机工程系，首尔国立大学（SNU）和自动化与系统研究所（ASRI）以及Sequor机器人公司）；Department of Electrical and Computer Engineering, Seoul National University (SNU) and Automation and Systems Research Institute (ASRI)（电气与计算机工程系，首尔国立大学（SNU）和自动化与系统研究所（ASRI））；Interdisciplinary Program in Artificial Intelligence, Seoul National University (SNU) and Automation and Systems Research Institute (ASRI) and Sequor Robotics Inc.（人工智能跨学科项目，首尔国立大学（SNU）和自动化与系统研究所（ASRI）以及Sequor机器人公司）；Sequor Robotics Inc.（Sequor机器人公司）
A Computer Vision Pipeline for Individual-Level Behavior Analysis: Benchmarking on the Edinburgh Pig Dataset
Department of Animal Science, College of Agriculture and Life Sciences, Cornell University（动物科学系，农业与生命科学学院，康奈尔大学）；University of Liège, Gembloux Agro-Bio Tech (ULìege-GxABT)（列日大学，吉尔伯特农业生物技术（ULiège-GxABT））
SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining
University of Amsterdam（阿姆斯特丹大学）；Computer Vision Lab, ETH Zurich（苏黎世联邦理工学院计算机视觉实验室）；INSAIT, Sofia University ”St. Kliment Ohridski”（索菲亚大学”圣克莱门特·欧赫里迪斯”研究所）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Pisa（比萨大学）；University of Trento（特伦特大学）
