The prediction of surrounding agents' motion is a key for safe autonomous driving. In this paper, we explore navigation maps as an alternative to the predominant High Definition (HD) maps for learning-based motion prediction. Navigation maps provide topological and geometrical information on road-level, HD maps additionally have centimeter-accurate lane-level information. As a result, HD maps are costly and time-consuming to obtain, while navigation maps with near-global coverage are freely available. We describe an approach to integrate navigation maps into learning-based motion prediction models. To exploit locally available HD maps during training, we additionally propose a model-agnostic method for knowledge distillation. In experiments on the publicly available Argoverse dataset with navigation maps obtained from OpenStreetMap, our approach shows a significant improvement over not using a map at all. Combined with our method for knowledge distillation, we achieve results that are close to the original HD map-reliant models. Our publicly available navigation map API for Argoverse enables researchers to develop and evaluate their own approaches using navigation maps.
Georgia Institute of Technology（佐治亚理工学院）；FAIR, Meta AI（元人工智能基础人工智能研究院）；Simon Fraser University（西蒙弗雷泽大学）；Google Research Atlanta（谷歌亚特兰大研究院）；Oregon State University（俄勒冈州立大学）
Animal navigation research posits that organisms build and maintain internal spatial representations, or maps, of their environment. We ask if machines -- specifically, artificial intelligence (AI) navigation agents -- also build implicit (or 'mental') maps. A positive answer to this question would (a) explain the surprising phenomenon in recent literature of ostensibly map-free neural-networks achieving strong performance, and (b) strengthen the evidence of mapping as a fundamental mechanism for navigation by intelligent embodied agents, whether they be biological or artificial. Unlike animal…
NavMapFusion: Diffusion-based Fusion of Navigation Maps for Online Vectorized HD Map Construction
Mercedes-Benz Research & Development North America, USA（梅赛德斯-奔驰北美研究与开发）；University of Stuttgart, Germany（斯图加特大学）；University of California, San Diego, USA（加州大学圣地亚哥分校）；University of Southampton, United Kingdom（南安普顿大学）
Facebook AI Research（Facebook人工智能研究院）；University of California, Berkeley（加州大学伯克利分校）；Georgia Institute of Technology（佐治亚理工学院）
We introduce a learning-based approach for room navigation using semantic maps. Our proposed architecture learns to predict top-down belief maps of regions that lie beyond the agent's field of view while modeling architectural and stylistic regularities in houses. First, we train a model to generate amodal semantic top-down maps indicating beliefs of location, size, and shape of rooms by learning the underlying architectural patterns in houses. Next, we use these maps to predict a point that lies in the target room and train a policy to navigate to the point. We empirically demonstrate that by…
InnoPeak Technology, Inc.（InnoPeak科技公司）；Monash University（莫纳什大学）；The University of Adelaide（阿德莱德大学）
We propose a robotic learning system for autonomous exploration and navigation in unexplored environments. We are motivated by the idea that even an unseen environment may be familiar from previous experiences in similar environments. The core of our method, therefore, is a process for building, predicting, and using probabilistic layout graphs for assisting goal-based visual navigation. We describe a navigation system that uses the layout predictions to satisfy high-level goals (e.g. "go to the kitchen") more rapidly and accurately than the prior art. Our proposed navigation framework compris…
University of Freiburg（弗赖堡大学）；Google Research（谷歌研究院）；University of Technology Nuremberg（纽伦堡工业大学）
While interacting in the world is a multi-sensory experience, many robots continue to predominantly rely on visual perception to map and navigate in their environments. In this work, we propose Audio-Visual-Language Maps (AVLMaps), a unified 3D spatial map representation for storing cross-modal information from audio, visual, and language cues. AVLMaps integrate the open-vocabulary capabilities of multimodal foundation models pre-trained on Internet-scale data by fusing their features into a centralized 3D voxel grid. In the context of navigation, we show that AVLMaps enable robot systems to i…
Navigation tasks in photorealistic 3D environments are challenging because they require perception and effective planning under partial observability. Recent work shows that map-like memory is useful for long-horizon navigation tasks. However, a focused investigation of the impact of maps on navigation tasks of varying complexity has not yet been performed. We propose the multiON task, which requires navigation to an episode-specific sequence of objects in a realistic environment. MultiON generalizes the ObjectGoal navigation task and explicitly tests the ability of navigation agents to locate…
Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs
Australian Institute for Machine Learning at the University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；School of Software Engineering, South China University of Technology（华南理工大学软件工程学院）
University of Chinese Academy of Sciences（中国科学院大学）；Monash University（莫纳什大学）；Sun Yat-sen University（中山大学）；Dark Matter AI Inc.（暗物质人工智能公司）
Vision-dialog navigation posed as a new holy-grail task in vision-language disciplinary targets at learning an agent endowed with the capability of constant conversation for help with natural language and navigating according to human responses. Besides the common challenges faced in visual language navigation, vision-dialog navigation also requires to handle well with the language intentions of a series of questions about the temporal context from dialogue history and co-reasoning both dialogs and visual scenes. In this paper, we propose the Cross-modal Memory Network (CMN) for remembering an…
University of Bonn（波恩大学）；Robert Bosch GmbH（罗伯特·博世有限公司）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）
We investigate the task of object goal navigation in unknown environments where the target is specified by a semantic label (e.g. find a couch). Such a navigation task is especially challenging as it requires understanding of semantic context in diverse settings. Most of the prior work tackles this problem under the assumption of a discrete action policy whereas we present an approach with continuous control which brings it closer to real world applications. We propose a deep neural network architecture and loss function to predict dense cost maps that implicitly contain semantic context and g…
IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation
Michigan State University（密歇根州立大学）；Independent Researcher（独立研究者）；University of California, Irvine（加州大学尔湾分校）；Ohio State University（俄亥俄州立大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of Pittsburgh（匹兹堡大学）；University of Virginia（弗吉尼亚大学）；Arizona State University（亚利桑那州立大学）；Purdue University Northwest（普渡大学西北分校）
Monash University（莫纳什大学）；University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-sen University（中山大学）；RMIT University（皇家墨尔本理工大学）
"Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This problem has attracted rising attention in recent years due to its wide application in autonomous driving, vacuum cleaner, and rescue robot. A navigation agent is supposed to have various intelligent skills, such as visual perceiving, mapping, planning, exploring and reasoning, etc. Building such an agent that observes, thinks, and acts is a key to real intelligence. The remarkable learning ability of deep learning methods empowered the agents to accomplish em…
Visual Indoor Navigation (VIN) task has drawn increasing attention from the data-driven machine learning communities especially with the recently reported success from learning-based methods. Due to the innate complexity of this task, researchers have tried approaching the problem from a variety of different angles, the full scope of which has not yet been captured within an overarching report. This survey first summarizes the representative work of learning-based approaches for the VIN task and then identifies and discusses lingering issues impeding the VIN performance, as well as motivates f…
This paper describes a system whereby a robot detects and track human-meaningful navigational cues as it navigates in an indoor environment. It is intended as the sensor front-end for a mobile robot system that can communicate its navigational context with human users. From simulated LiDAR scan data we construct a set of 2D occupancy grid bitmaps, then hand-label these with human-scale navigational features such as closed doors, open corridors and intersections. We train a Convolutional Neural Network (CNN) to recognize these features on input bitmaps. In our demonstration system, these featur…
In video games, non-player characters (NPCs) are used to enhance the players' experience in a variety of ways, e.g., as enemies, allies, or innocent bystanders. A crucial component of NPCs is navigation, which allows them to move from one point to another on the map. The most popular approach for NPC navigation in the video game industry is to use a navigation mesh (NavMesh), which is a graph representation of the map, with nodes and edges indicating traversable areas. Unfortunately, complex navigation abilities that extend the character's capacity for movement, e.g., grappling hooks, jetpacks…
We present the concept of speed maps: speed limits for mobile robots in human environments. Static speed maps allow for faster navigation on corridors while limiting the speed around corners and in rooms. Dynamic speed maps put limits on speed around humans. We demonstrate the concept for a mobile robot that guides people to annotated landmarks on the map. The robot keeps a metric map for navigation and a semantic map to hold planar surfaces for tasking. The system supports automatic initialization upon the detection of a specially designed QR code. We show that speed maps not only can reduce …
Norwegian University of Life Sciences（挪威生命科学大学）；University of Lincoln（林肯大学）
This paper introduces a new method for robot motion planning and navigation in uneven environments through a surfel representation of underlying point clouds. The proposed method addresses the shortcomings of state-of-the-art navigation methods by incorporating both kinematic and physical constraints of a robot with standard motion planning algorithms (e.g., those from the Open Motion Planning Library), thus enabling efficient sampling-based planners for challenging uneven terrain navigation on raw point cloud maps. Unlike techniques based on Digital Elevation Maps (DEMs), our novel surfel-bas…
University of Pennsylvania（宾夕法尼亚大学）；The University of Chicago（芝加哥大学）
We consider the problems of exploration and point-goal navigation in previously unseen environments, where the spatial complexity of indoor scenes and partial observability constitute these tasks challenging. We argue that learning occupancy priors over indoor maps provides significant advantages towards addressing these problems. To this end, we present a novel planning framework that first learns to generate occupancy maps beyond the field-of-view of the agent, and second leverages the model uncertainty over the generated areas to formulate path selection policies for each task of interest. …
Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments. Inspired by the robotics community, we propose a modular approach to VLN using topological maps. Given a natural language instruction and topological map, our approach leverages attention mechanisms to predict a navigation plan in the map. The plan is then executed with low-level actions (e.g. forward, rotate) using a robust controller. Experiments show that our method outperforms previous end-to-end approaches, generates interpretable navigati…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook 人工智能研究院）
State-of-the-art navigation methods leverage a spatial memory to generalize to new environments, but their occupancy maps are limited to capturing the geometric structures directly observed by the agent. We propose occupancy anticipation, where the agent uses its egocentric RGB-D observations to infer the occupancy state beyond the visible regions. In doing so, the agent builds its spatial awareness more rapidly, which facilitates efficient exploration and navigation in 3D environments. By exploiting context in both the egocentric views and top-down maps our model successfully anticipates a br…
Université de Nantes（南特大学）；Beijing University of Technology（北京工业大学）
Free-Viewpoint Video (FVV) systems allow the viewers to freely change the viewpoints of the scene. In such systems, view synthesis and compression are the two main sources of artifacts influencing the perceived quality. To assess this influence, quality evaluation studies are often carried out using conventional displays and generating predefined navigation trajectories mimicking the possible movement of the viewers when exploring the content. Nevertheless, as different trajectories may lead to different conclusions in terms of visual quality when benchmarking the performance of the systems, m…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
In this work, we have implemented a SLAM-assisted navigation module for a real autonomous vehicle with unknown dynamics. The navigation objective is to reach a desired goal configuration along a collision-free trajectory while adhering to the dynamics of the system. Specifically, we use LiDAR-based Hector SLAM for building the map of the environment, detecting obstacles, and for tracking vehicle's conformance to the trajectory as it passes through various states. For motion planning, we use rapidly exploring random trees (RRTs) on a set of generated motion primitives to search for dynamically …
Exploring Unstructured Environments using Minimal Sensing on Cooperative Nano-Drones
Computer Vision and Aerial Robotics Group at Universidad Politécnica de Madrid（马德里理工大学计算机视觉与空中机器人小组）；Centre for Automation and Robotics C.A.R. (UPM-CSIC)（自动化与机器人中心（UPM-CSIC））；Department of Mechanical Engineering, Texas A&M University（机械工程系，德克萨斯A&M大学）
University of Freiburg（弗赖堡大学）；Google Research（谷歌研究院）；University of Technology Nuremberg（纽伦堡工业大学）
Grounding language to the visual observations of a navigating agent can be performed using off-the-shelf visual-language models pretrained on Internet-scale data (e.g., image captions). While this is useful for matching images to natural language descriptions of object goals, it remains disjoint from the process of mapping the environment, so that it lacks the spatial precision of classic geometric maps. To address this problem, we propose VLMaps, a spatial map representation that directly fuses pretrained visual-language features with a 3D reconstruction of the physical world. VLMaps can be a…
Instance-Enriched Semantic Maps for Visual Language Navigation
Department of Mechanical Engineering, Kyung Hee University（庆熙大学机械工程系）；Advanced Institutes of Convergence Technology (AICT)（融合技术高级研究院）；School of Mechanical Engineering, Kyungpook National University（庆北国立大学机械工程学院）
IIIT Hyderabad（印度海得拉巴国际信息技术学院）；Intel Bangalore（英特尔班加罗尔分公司）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
While several datasets for autonomous navigation have become available in recent years, they tend to focus on structured driving environments. This usually corresponds to well-delineated infrastructure such as lanes, a small number of well-defined categories for traffic participants, low variation in object or background appearance and strict adherence to traffic rules. We propose IDD, a novel dataset for road scene understanding in unstructured environments where the above assumptions are largely not satisfied. It consists of 10,004 images, finely annotated with 34 classes collected from 182 …
FOM-Nav: Frontier-Object Maps for Object Goal Navigation
Inria（法国国家信息与自动化技术研究院）；Department of Computer Science, École normale supérieure (ENS-PSL, CNRS, Inria)（高等师范学院计算机科学系）；Courant Institute of Mathematical Sciences and Center for Data Science, New York University（纽约大学Courant数学科学研究所和数据科学中心）
MLFM: Multi-Layered Feature Maps for Richer Language Understanding in Zero-Shot Semantic Navigation
Simon Fraser University（西蒙弗雷泽大学）；University of Padova（帕多瓦大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·克塞勒基金会）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔人工智能研究所）
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学）；INRIA（法国国家信息与自动化研究所）
This paper addresses the problem of building augmented metric representations of scenes with semantic information from RGB-D images. We propose a complete framework to create an enhanced map representation of the environment with object-level information to be used in several applications such as human-robot interaction, assistive robotics, visual navigation, or in manipulation tasks. Our formulation leverages a CNN-based object detector (Yolo) with a 3D model-based segmentation technique to perform instance semantic segmentation, and to localize, identify, and track different classes of objec…
Terrain assessment is a key aspect for autonomous exploration rovers, surrounding environment recognition is required for multiple purposes, such as optimal trajectory planning and autonomous target identification. In this work we present a technique to generate accurate three-dimensional semantic maps for Martian environment. The algorithm uses as input a stereo image acquired by a camera mounted on a rover. Firstly, images are labeled with DeepLabv3+, which is an encoder-decoder Convolutional Neural Networl (CNN). Then, the labels obtained by the semantic segmentation are combined to stereo …
