我们提出了VLPG-Nav，一种用于引导机器人在家庭场景中找到指定目标的视觉语言导航方法。现有方法主要聚焦于让机器人向目标导航，而我们的方法额外考虑了将目标置于机器人相机视野中心的挑战。我们的方法构建了视觉语言位姿图（VLPG，visual language pose graph），它作为视觉语言嵌入的空间地图。给定一个开放词汇的目标查询，我们利用VLPG为目标导航规划一个视点。即便导航到了该视点，目标遮挡、移位以及机器人定位误差等现实挑战仍可能导致目标不可见。我们构建了目标定位概率图，它利用机器人当前的观测结果和先验VLPG。当目标不可见时，概率图会更新，并计算出备选视点。此外，我们提出了一种目标居中公式，可局部调整机器人位姿，使目标处于相机视野中心。我们通过仿真和真实世界实验评估了该方法的有效性，评估了其成功在相机视野中观测到目标并将目标居中的能力。VLPG-Nav在目标定位、绕开遮挡导航以及将目标置于机器人相机视野中心方面表现出更优的性能，在评估指标上优于所选的基准方法。
英文摘要：
We present VLPG-Nav, a visual language navigation method for guiding robots to specified objects within household scenes. Unlike existing methods primarily focused on navigating the robot toward objects, our approach considers the additional challenge of centering the object within the robot's camera view. Our method builds a visual language pose graph (VLPG) that functions as a spatial map of VL embeddings. Given an open vocabulary object query, we plan a viewpoint for object navigation using the VLPG. Despite navigating to the viewpoint, real-world challenges like object occlusion, displacement, and the robot's localization error can prevent visibility. We build an object localization probability map that leverages the robot's current observations and prior VLPG. When the object isn't visible, the probability map is updated and an alternate viewpoint is computed. In addition, we propose an object-centering formulation that locally adjusts the robot's pose to center the object in the camera view. We evaluate the effectiveness of our approach through simulations and real-world experiments, evaluating its ability to successfully view and center the object within the camera field of view. VLPG-Nav demonstrates improved performance in locating the object, navigating around occlusions, and centering the object within the robot's camera view, outperforming the selected baselines in the evaluation metrics.
Samsung Research（三星研究院）；Rey Juan Carlos University（胡安卡洛斯国王大学）
Developments in mobile robot navigation have enabled robots to operate in warehouses, retail stores, and on sidewalks around pedestrians. Various navigation solutions have been proposed, though few as widely adopted as ROS Navigation. 10 years on, it is still one of the most popular navigation solutions. Yet, ROS Navigation has failed to keep up with modern trends. We propose the new navigation solution, Navigation2, which builds on the successful legacy of ROS Navigation. Navigation2 uses a behavior tree for navigator task orchestration and employs new methods designed for dynamic environment…
Zhejiang University（浙江大学）；The University of Technology, Sydney（悉尼科技大学）
Autonomous navigation based on precise localization has been widely developed in both academic research and practical applications. The high demand for localization accuracy has been essential for safe robot planing and navigation while it makes the current geometric solutions less robust to environmental changes. Recent research on end-to-end methods handle raw sensory data with forms of navigation instructions and directly output the command for robot control. However, the lack of intermediate semantics makes the system more rigid and unstable for practical use. To explore these issues, this…
We demonstrated multi-mobile robot navigation based on Visible Light Positioning(VLP) localization. From our experiment, the VLP can accurately locate robots' positions in navigation.
Recent applications of deep learning to navigation have generated end-to-end navigation solutions whereby visual sensor input is mapped to control signals or to motion primitives. The resulting visual navigation strategies work very well at collision avoidance and have performance that matches traditional reactive navigation algorithms while operating in real-time. It is accepted that these solutions cannot provide the same level of performance as a global planner. However, it is less clear how such end-to-end systems should be integrated into a full navigation pipeline. We evaluate the typica…
Anhui University（安徽大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Maryland, College Park（马里兰大学帕克分校）
We present a novel approach for image-goal navigation, where an agent navigates with a goal image rather than accurate target information, which is more challenging. Our goal is to decouple the learning of navigation goal planning, collision avoidance, and navigation ending prediction, which enables more concentrated learning of each part. This is realized by four different modules. The first module maintains an obstacle map during robot navigation. The second predicts a long-term goal on the real-time map periodically, which can thus convert an image-goal navigation task to several point-goal…
Multi-robot navigation is a challenging task in which multiple robots must be coordinated simultaneously within dynamic environments. We apply deep reinforcement learning (DRL) to learn a decentralized end-to-end policy which maps raw sensor data to the command velocities of the agent. In order to enable the policy to generalize, the training is performed in different environments and scenarios. The learned policy is tested and evaluated in common multi-robot scenarios like switching a place, an intersection and a bottleneck situation. This policy allows the agent to recover from dead ends and…
Towards bio-inspired visual navigation: a neuromorphic navigation paradigm for MAVs
Faculty of Robot Science and Engineering, Northeastern University（东北大学机器人科学与工程学院）；College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）
Robotics Institute（机器人研究所）；University of Technology Sydney（悉尼科技大学）
Underwater visual-inertial navigation is challenging due to the poor visibility and presence of outliers in underwater environments. The navigation performance is closely related to outlier detection and elimination. Existing methods assume the inertial odometry is accurate enough for outlier detection, which is not valid for low-cost inertial applications. We propose a novel iterative smoothing and outlier detection method aiming for underwater navigation. Using the dataset collected from an underwater robot and fiducial markers, experimental results confirm that the method can successfully e…
The ROS navigation stack is powerful for mobile robots to move from place to place reliably. The job of navigation stack is to produce a safe path for the robot to execute, by processing data from odometry, sensors and environment map. Maximizing the performance of this navigation stack requires some fine tuning of parameters, and this is not as simple as it looks. One who is sophomoric about the concepts and reasoning may try things randomly, and wastes a lot of time. This article intends to guide the reader through the process of fine tuning navigation parameters. It is the reference when so…
Most current approaches to social navigation focus on the trajectory and position of participants in the interaction. Our current work on the topic focuses on integrating gaze into social navigation, both to cue nearby pedestrians as to the intended trajectory of the robot and to enable the robot to read the intentions of nearby pedestrians. This paper documents a series of experiments in our laboratory investigating the role of gaze in social navigation.
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）
Learning from Hallucination (LfH) is a recent machine learning paradigm for autonomous navigation, which uses training data collected in completely safe environments and adds numerous imaginary obstacles to make the environment densely constrained, to learn navigation planners that produce feasible navigation even in highly constrained (more dangerous) spaces. However, LfH requires hallucinating the robot perception during deployment to match with the hallucinated training data, which creates a need for sometimes-infeasible prior knowledge and tends to generate very conservative planning. In t…
Towards Path-Creative Navigation: Robot Navigation through Embodied Interaction
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；College of Information Science and Technology, Eastern Institute of Technology（东方理工大学信息科学与技术学院）；School of Mechanical and Power Engineering, Zhengzhou university（郑州大学机械与动力工程学院）；University of Waterloo（滑铁卢大学）
Popular navigation stacks implemented on top of open-source frameworks such as ROS(Robot Operating System) and ROS2 represent the robot workspace using a discretized 2D occupancy grid. This method, while requiring less computation, restricts the use of such navigation stacks to wheeled robots navigating on flat surfaces. In this paper, we present a navigation stack that uses a volumetric representation of the robot workspace, and hence can be extended to aerial and legged robots navigating through uneven terrain. Additionally, we present a new sampling-based motion planning algorithm which int…
EnQuery: Ensemble Policies for Diverse Query-Generation in Preference Alignment of Robot Navigation
University of Bonn（波恩大学）；Center for Robotics at the University of Bonn（波恩大学机器人中心）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）
Deep reinforcement learning (DRL) demonstrates its potential in learning a model-free navigation policy for robot visual navigation. However, the data-demanding algorithm relies on a large number of navigation trajectories in training. Existing datasets supporting training such robot navigation algorithms consist of either 3D synthetic scenes or reconstructed scenes. Synthetic data suffers from domain gap to the real-world scenes while visual inputs rendered from 3D reconstructed scenes have undesired holes and artifacts. In this paper, we present a new dataset collected in real-world to facil…
West Virginia University（西弗吉尼亚大学）；The Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）
In this work, we demonstrate the importance of zero velocity information for global navigation satellite system (GNSS) based navigation. The effectiveness of using the zero velocity information with zero velocity update (ZUPT) for inertial navigation applications have been shown in the literature. Here we leverage this information and add it as a position constraint in a GNSS factor graph. We also compare its performance to a GNSS/inertial navigation system (INS) coupled factor graph. We tested our ZUPT aided factor graph method on three datasets and compared it with the GNSS-only factor graph…
Ohio State University（俄亥俄州立大学）；Ottonomy Inc.（Ottonomy公司）
The popularity of mobile robots has been steadily growing, with these robots being increasingly utilized to execute tasks previously completed by human workers. For bipedal robots to see this same success, robust autonomous navigation systems need to be developed that can execute in real-time and respond to dynamic environments. These systems can be divided into three stages: perception, planning, and control. A holistic navigation framework for bipedal robots must successfully integrate all three components of the autonomous navigation problem to enable robust real-world navigation. In this p…
Robots moving safely and in a socially compliant manner in dynamic human environments is an essential benchmark for long-term robot autonomy. However, it is not feasible to learn and benchmark social navigation behaviors entirely in the real world, as learning is data-intensive, and it is challenging to make safety guarantees during training. Therefore, simulation-based benchmarks that provide abstractions for social navigation are required. A framework for these benchmarks would need to support a wide variety of learning approaches, be extensible to the broad range of social navigation scenar…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；National University of Defense Technology（国防科技大学）；Zhengzhou University（郑州大学）；University of Maryland, College Park（马里兰大学帕克分校）
To enhance the cross-target and cross-scene generalization of target-driven visual navigation based on deep reinforcement learning (RL), we introduce an information-theoretic regularization term into the RL objective. The regularization maximizes the mutual information between navigation actions and visual observation transforms of an agent, thus promoting more informed navigation decisions. This way, the agent models the action-observation dynamics by learning a variational generative model. Based on the model, the agent generates (imagines) the next observation from its current observation a…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）
This paper presents a self-improving lifelong learning framework for a mobile robot navigating in different environments. Classical static navigation methods require environment-specific in-situ system adjustment, e.g. from human experts, or may repeat their mistakes regardless of how many times they have navigated in the same environment. Having the potential to improve with experience, learning-based navigation is highly dependent on access to training resources, e.g. sufficient memory and fast computation, and is prone to forgetting previously learned capability, especially when facing diff…
Robots navigating in human crowds need to optimize their paths not only for their task performance but also for their compliance to social norms. One of the key challenges in this context is the lack of standard metrics for evaluating and optimizing a socially compliant behavior. Existing works in social navigation can be grouped according to the differences in their optimization objectives. For instance, the reinforcement learning approaches tend to optimize on the \textit{comfort} aspect of the socially compliant navigation, whereas the inverse reinforcement learning approaches are designed …
LERR-Nav: Learning to Coordinate Exploration, Recovery, and Reminiscing for Object Navigation
Hong Kong Polytechnic University（香港理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；DeepRobotics（深度机器人公司）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangzhou University（广州大学）
We present EWareNet, a novel intent and affect-aware social robot navigation algorithm among pedestrians. Our approach predicts the trajectory-based pedestrian intent from gait sequence, which is then used for intent-guided navigation taking into account social and proxemic constraints. We propose a transformer-based model that works on commodity RGB-D cameras mounted onto a moving robot. Our intent prediction routine is integrated into a mapless navigation scheme and makes no assumptions about the environment of pedestrian motion. Our navigation scheme consists of a novel obstacle profile rep…
Metric ground navigation addresses the problem of autonomously moving a robot from one point to another in an obstacle-occupied planar environment in a collision-free manner. It is one of the most fundamental capabilities of intelligent mobile robots. This paper presents a standardized testbed with a set of environments and metrics to benchmark difficulty of different scenarios and performance of different systems of metric ground navigation. Current benchmarks focus on individual components of mobile robot navigation, such as perception and state estimation, but the navigation performance as …
Cyprus University of Technology（塞浦路斯理工大学）；Technion Israel Institute of Technology（以色列理工学院）
This paper brings together the concepts of navigation transformation and harmonic functions to form navigation functions that are correct-by-construction in the sense that no tuning is required. The form of the navigation function is explicitly related to the number of obstacles in the environment. This enables application of navigation functions for autonomous robot navigation in partially or fully unknown environments, with the capability of on-the-fly adjustment of the navigation function when new obstacles are discovered by the robot. Appropriate navigation controllers, applicable to robot…
Intelligent navigation among social crowds is an essential aspect of mobile robotics for applications such as delivery, health care, or assistance. Deep Reinforcement Learning emerged as an alternative planning method to conservative approaches and promises more efficient and flexible navigation. However, in highly dynamic environments employing different kinds of obstacle classes, safe navigation still presents a grand challenge. In this paper, we propose a semantic Deep-reinforcement-learning-based navigation approach that teaches object-specific safety rules by considering high-level obstac…
InnoPeak Technology, Inc.（InnoPeak科技公司）；Monash University（莫纳什大学）；The University of Adelaide（阿德莱德大学）
We propose a robotic learning system for autonomous exploration and navigation in unexplored environments. We are motivated by the idea that even an unseen environment may be familiar from previous experiences in similar environments. The core of our method, therefore, is a process for building, predicting, and using probabilistic layout graphs for assisting goal-based visual navigation. We describe a navigation system that uses the layout predictions to satisfy high-level goals (e.g. "go to the kitchen") more rapidly and accurately than the prior art. Our proposed navigation framework compris…
This work proposes a novel transformation termed the conformal navigation transformation to achieve collision-free navigation of a robot in a workspace populated with arbitrary polygonal obstacles. The properties of the conformal navigation transformation in the polygonal workspace are investigated in this work as well as its capability to provide a solution to the navigation problem. %The properties of the conformal navigation transformation are investigated, which contribute to the solution of the robot navigation problem in complex polygonal environments. %which facilitates the navigation o…
Human navigation in built environments depends on symbolic spatial information which has unrealised potential to enhance robot navigation capabilities. Information sources such as labels, signs, maps, planners, spoken directions, and navigational gestures communicate a wealth of spatial information to the navigators of built environments; a wealth of information that robots typically ignore. We present a robot navigation system that uses the same symbolic spatial information employed by humans to purposefully navigate in unseen built environments with a level of performance comparable to human…
This paper presents a novel end-to-end Unmanned Aerial System (UAS) navigation approach for long-range visual navigation in the real world. Inspired by dual-process visual navigation system of human's instinct: environment understanding and landmark recognition, we formulate the UAS navigation task into two same phases. Our system combines the reinforcement learning (RL) and image matching approaches. First, the agent learns the navigation policy using RL in the specified environment. To achieve this, we design an interactive UASNAV environment for the training process. Once the agent learns t…
Pontificia Universidad Catolica de Chile（智利天主教大学）；Stanford University（斯坦福大学）
We present a semantically rich graph representation for indoor robotic navigation. Our graph representation encodes: semantic locations such as offices or corridors as nodes, and navigational behaviors such as enter office or cross a corridor as edges. In particular, our navigational behaviors operate directly from visual inputs to produce motor controls and are implemented with deep learning architectures. This enables the robot to avoid explicit computation of its precise location or the geometry of the environment, and enables navigation at a higher level of semantic abstraction. We evaluat…
Northeastern University（东北大学）；Ocean University of China（中国海洋大学）
Navigation functions provide both path and motion planning, which can be used to ensure obstacle avoidance and convergence in the sphere world. When dealing with complex and realistic scenarios, constructing a transformation to the sphere world is essential and, at the same time, challenging. This work proposes a novel transformation termed the conformal navigation transformation to achieve collision-free navigation of a robot in a workspace populated with obstacles of arbitrary shapes. The properties of the conformal navigation transformation, including uniqueness, invariance of navigation pr…
The primary objective of a safe navigation algorithm is to guide the object from its current position to the target position while avoiding any collision with the en-route obstacles, and the appropriate obstacle avoidance strategies are the key factors to ensure safe navigation tasks in dynamic environments. In this report, three different obstacle avoidance strategies for safe navigation in dynamic environments have been presented. The biologically-inspired navigation algorithm (BINA) is efficient in terms of avoidance time. The equidistant based navigation algorithm (ENA) is able to achieve …
