When navigating in a man-made environment they haven't visited before--like an office building--humans employ behaviors such as reading signs and asking others for directions. These behaviors help humans reach their destinations efficiently by reducing the need to search through large areas. Existing robot navigation systems lack the ability to execute such behaviors and are thus highly inefficient at navigating within large environments. We present ReasonNav, a modular navigation system which integrates these human-like navigation skills by leveraging the reasoning capabilities of a vision-language model (VLM). We design compact input and output abstractions based on navigation landmarks, allowing the VLM to focus on language understanding and reasoning. We evaluate ReasonNav on real and simulated navigation tasks and show that the agent successfully employs higher-order reasoning to navigate efficiently in large, complex buildings.
Human navigation in built environments depends on symbolic spatial information which has unrealised potential to enhance robot navigation capabilities. Information sources such as labels, signs, maps, planners, spoken directions, and navigational gestures communicate a wealth of spatial information to the navigators of built environments; a wealth of information that robots typically ignore. We present a robot navigation system that uses the same symbolic spatial information employed by humans to purposefully navigate in unseen built environments with a level of performance comparable to human…
Hong Kong University of Science and Technology（香港科技大学）
Safe and efficient crowd navigation for mobile robot is a crucial yet challenging task. Previous work has shown the power of deep reinforcement learning frameworks to train efficient policies. However, their performance deteriorates when the crowd size grows. We suggest that this can be addressed by enabling the network to identify and pay attention to the humans in the crowd that are most critical to navigation. We propose a novel network utilizing a graph representation to learn the policy. We first train a graph convolutional network based on human gaze data that accurately predicts human a…
University of Bonn（波恩大学）；Fraunhofer FKIE（弗劳恩霍夫通信、信息处理与人机工程研究所）
For the most comfortable, human-aware robot navigation, subjective user preferences need to be taken into account. This paper presents a novel reinforcement learning framework to train a personalized navigation controller along with an intuitive virtual reality demonstration interface. The conducted user study provides evidence that our personalized approach significantly outperforms classical approaches with more comfortable human-robot experiences. We achieve these results using only a few demonstration trajectories from non-expert users, who predominantly appreciate the intuitive demonstrat…
ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles
Department of Aerospace Engineering, Seoul National University（首尔国立大学航空航天工程系）；InnoCORE AI-Transformed Aerospace Research Center, KAIST（韩国科学技术院（KAIST）人工智能赋能航空航天研究中心）
Simon Fraser University（西蒙弗雷泽大学）；School of Computing Science（计算机科学学院）；Burnaby, BC Canada（加拿大不列颠哥伦比亚省本拿比）
This article describes a dataset collected in a set of experiments that involves human participants and a robot. The set of experiments was conducted in the computing science robotics lab in Simon Fraser University, Burnaby, BC, Canada, and its aim is to gather data containing common gestures, movements, and other behaviours that may indicate humans' navigational intent relevant for autonomous robot navigation. The experiment simulates a shopping scenario where human participants come in to pick up items from his/her shopping list and interact with a Pepper robot that is programmed to help the…
University of California San Diego（加州大学圣地亚哥分校）；Monash University（莫纳什大学）
We discuss the process of building semantic maps, how to interactively label entities in them, and how to use them to enable context-aware navigation behaviors in human environments. We utilize planar surfaces, such as walls and tables, and static objects, such as door signs, as features for our semantic mapping approach. Users can interactively annotate these features by having the robot follow him/her, entering the label through a mobile app, and performing a pointing gesture toward the landmark of interest. Our gesture based approach can reliably estimate which object is being pointed at an…
LAAS-CNRS（法国国家科学研究中心系统分析与架构实验室）；CNRS（法国国家科学研究中心）；Artificial and Natural Intelligence Toulouse Institute (ANITI)（图卢兹人工与自然智能研究所）
As more robots are being deployed into human environments, a human-aware navigation planner needs to handle multiple contexts that occur in indoor and outdoor environments. In this paper, we propose a tunable human-aware robot navigation planner that can handle a variety of humanrobot contexts. We present the architecture of the planner and discuss the features and some implementation details. Then we present a detailed analysis of various simulated humanrobot contexts using the proposed planner along with some quantitative results. Finally, we show the results in a real-world scenario after d…
LAAS-CNRS, Universit de Toulouse, CNRS（法国国家科学研究中心系统分析与架构实验室，图卢兹大学，法国国家科学研究中心）
We claim that navigation in human environments can be viewed as cooperative activity especially in constrained situations. Humans concurrently aid and comply with each other while moving in a shared space. Cooperation helps pedestrians to efficiently reach their own goals and respect conventions such as the personal space of others. To meet human comparable efficiency, a robot needs to predict the human trajectories and plan its own trajectory correspondingly in the same shared space. In this work, we present a navigation planner that is able to plan such cooperative trajectories, simultaneous…
The purpose of navigation is to determine the position, velocity, and orientation of manned and autonomous platforms, humans, and animals. Obtaining accurate navigation commonly requires fusion between several sensors, such as inertial sensors and global navigation satellite systems, in a model-based, nonlinear estimation framework. Recently, data-driven approaches applied in various fields show state-of-the-art performance, compared to model-based methods. In this paper we review multidisciplinary, data-driven based navigation algorithms developed and experimentally proven at the Autonomous N…
Humans are known to construct cognitive maps of their everyday surroundings using a variety of perceptual inputs. As such, when a human is asked for directions to a particular location, their wayfinding capability in converting this cognitive map into directional instructions is challenged. Owing to spatial anxiety, the language used in the spoken instructions can be vague and often unclear. To account for this unreliability in navigational guidance, we propose a novel Deep Reinforcement Learning (DRL) based trust-driven robot navigation algorithm that learns humans' trustworthiness to perform…
University of Catania（卡塔尼亚大学）；Philips Research（飞利浦研究院）；ICAR-CNR（意大利国家研究委员会信息与高性能计算与网络研究所）
Intelligent systems are increasingly part of our everyday lives and have been integrated seamlessly to the point where it is difficult to imagine a world without them. Physical manifestations of those systems on the other hand, in the form of embodied agents or robots, have so far been used only for specific applications and are often limited to functional roles (e.g. in the industry, entertainment and military fields). Given the current growth and innovation in the research communities concerned with the topics of robot navigation, human-robot-interaction and human activity recognition, it se…
Risky and crowded environments (RCE) contain abstract sources of risk and uncertainty, which are perceived differently by humans, leading to a variety of behaviors. Thus, robots deployed in RCEs, need to exhibit diverse perception and planning capabilities in order to interpret other human agents' behavior and act accordingly in such environments. To understand this problem domain, we conducted a study to explore human path choices in RCEs, enabling better robotic navigational explainable AI (XAI) designs. We created a novel COVID-19 pandemic grocery shopping scenario which had time-risk trade…
LAAS-CNRS（法国国家科学研究中心自动化与系统分析实验室）；Universite de Toulouse（图卢兹大学）；CNRS（法国国家科学研究中心）
Current approaches in human-aware or social robot navigation address the humans that are visible to the robot. However, it is also important to address the possible emergences of humans to avoid shocks or surprises to humans and erratic behavior of the robot planner. In this paper, we propose a novel approach to detect and address these human emergences called `invisible humans'. We determine the places from which a human, currently not visible to the robot, can appear suddenly and then adapt the path and speed of the robot with the anticipation of potential collisions. This is done while stil…
We present SocialGym 2, a multi-agent navigation simulator for social robot research. Our simulator models multiple autonomous agents, replicating real-world dynamics in complex environments, including doorways, hallways, intersections, and roundabouts. Unlike traditional simulators that concentrate on single robots with basic kinematic constraints in open spaces, SocialGym 2 employs multi-agent reinforcement learning (MARL) to develop optimal navigation policies for multiple robots with diverse, dynamic constraints in complex environments. Built on the PettingZoo MARL library and Stable Basel…
Human-Aware Robot Navigation via Reinforcement Learning with Hindsight Experience Replay and Curriculum Learning
The Chinese University of Hong Kong（香港中文大学）；Southern University of Science and Technology（南方科技大学）；Shenzhen Research Institute of the Chinese University of Hong Kong（香港中文大学深圳研究院）
University of Extremadura（埃斯特雷马杜拉大学）；Aston University（阿斯顿大学）；Pune Institute of Computer Technology（浦那计算机技术学院）
Autonomous navigation is a key skill for assistive and service robots. To be successful, robots have to minimise the disruption caused to humans while moving. This implies predicting how people will move and complying with social conventions. Avoiding disrupting personal spaces, people's paths and interactions are examples of these social conventions. This paper leverages Graph Neural Networks to model robot disruption considering the movement of the humans and the robot so that the model built can be used by path planning algorithms. Along with the model, this paper presents an evolution of t…
We present EWareNet, a novel intent and affect-aware social robot navigation algorithm among pedestrians. Our approach predicts the trajectory-based pedestrian intent from gait sequence, which is then used for intent-guided navigation taking into account social and proxemic constraints. We propose a transformer-based model that works on commodity RGB-D cameras mounted onto a moving robot. Our intent prediction routine is integrated into a mapless navigation scheme and makes no assumptions about the environment of pedestrian motion. Our navigation scheme consists of a novel obstacle profile rep…
In this paper, we present a planning framework that uses a combination of implicit (robot motion) and explicit (visual/audio/haptic feedback) communication during mobile robot navigation. First, we developed a model that approximates both continuous movements and discrete behavior modes in human navigation, considering the effects of implicit and explicit communication on human decision making. The model approximates the human as an optimal agent, with a reward function obtained through inverse reinforcement learning. Second, a planner uses this model to generate communicative actions that max…
Tsinghua University（清华大学）；University of Bath（巴斯大学）
Pointing gestures are widely used in robot navigationapproaches nowadays. However, most approaches only use point-ing gestures, and these have two major limitations. Firstly, they need to recognize pointing gestures all the time, which leads to long processing time and significant system overheads. Secondly,the user's pointing direction may not be very accurate, so the robot may go to an undesired place. To relieve these limitations,we propose a voice-guided pointing robot navigation approach named VGPN, and implement its prototype on a wheeled robot,TurtleBot 2. VGPN recognizes a pointing ges…
Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues
Polytechnic of Turin（都灵理工大学）；University of Verona（维罗纳大学）；ENSTA Paris（巴黎国立高等先进技术学校）；Institut Polytechnique de Paris（巴黎综合理工学院）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）
Robots that can effectively understand human intentions from actions are crucial for successful human-robot collaboration. In this work, we address the challenge of a robot navigating towards an unknown goal while also accounting for a human's preference for a particular path in the presence of obstacles. This problem is particularly challenging when both the goal and path preference are unknown a priori. To overcome this challenge, we propose a method for encoding and inferring path preference online using a partitioning of the space into polytopes. Our approach enables joint inference over t…
In this paper, we present a framework for real-time autonomous robot navigation based on cloud and on-demand databases to address two major issues of human-like robot interaction and task planning in global dynamic environment, which is not known a priori. Our framework contributes to make human-like brain GPS mapping system for robot using spatial information and performs 3D visual semantic SLAM for independent robot navigation. We accomplish the feat by separating robot's memory system into Long-Term Memory (LTM) and Short-Term Memory (STM). We also form robot's behavior and knowledge system…
Imitation learning holds the promise to address challenging robotic tasks such as autonomous navigation. It however requires a human supervisor to oversee the training process and send correct control commands to robots without feedback, which is always prone to error and expensive. To minimize human involvement and avoid manual labeling of data in the robotic autonomous navigation with imitation learning, this paper proposes a novel semi-supervised imitation learning solution based on a multi-sensory design. This solution includes a suboptimal sensor policy based on sensor fusion to automatic…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Lab（约翰斯·霍普金斯大学应用物理实验室）；UC Berkeley（加州大学伯克利分校）
Human-aware robot navigation promises a range of applications in which mobile robots bring versatile assistance to people in common human environments. While prior research has mostly focused on modeling pedestrians as independent, intentional individuals, people move in groups; consequently, it is imperative for mobile robots to respect human groups when navigating around people. This paper explores learning group-aware navigation policies based on dynamic group formation using deep reinforcement learning. Through simulation experiments, we show that group-aware policies, compared to baseline…
The aim of this work is to build a cognitive model for the humanoid robot, especially, we are interested in the navigation and mapping on the humanoid robot. The agents used are the Alderbaran NAO robot. The framework is effectively applied to the integration of AI, computer vision, and signal processing problems. Our model can be divided into two parts, cognitive mapping and perception. Cognitive mapping is assumed as three parts, whose representations were proposed a network of ASRs, an MFIS, and a hierarchy of Place Representations. On the other hand, perception is the traditional computer …
Reinforcement Learning with Data Bootstrapping for Dynamic Subgoal Pursuit in Humanoid Robot Navigation
Mechanical and Aerospace Engineering, The Ohio State University（机械与航空航天工程，俄亥俄州立大学）；Electrical and Computer Engineering, The Ohio State University（电气与计算机工程，俄亥俄州立大学）；Computer Science and Engineering, The Ohio State University（计算机科学与工程，俄亥俄州立大学）
In this paper we investigate potential issues of fairness related to the motion of mobile robots. We focus on the particular use case of humanitarian mapping and disaster response. We start by showing that there is a fairness dimension to robot navigation, and use a walkthrough example to bring out design choices and issues that arise during the development of a fair system. We discuss indirect discrimination, fairness-efficiency trade-offs, the existence of counter-productive fairness definitions, privacy and other issues. Finally, we conclude with a discussion of the potential of our methodo…
University of Birmingham（伯明翰大学）；University of West Attica（西阿提卡大学）
This paper addresses the problem of human operator intent recognition during teleoperated robot navigation. In this context, recognition of the operator's intended navigational goal, could enable an artificial intelligence (AI) agent to assist the operator in an advanced human-robot interaction framework. We propose a Bayesian Operator Intent Recognition (BOIR) probabilistic method that utilizes: (i) an observation model that fuses information as a weighting combination of multiple observation sources providing geometric information; (ii) a transition model that indicates the evolution of the …
Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments
Department of Computer Science, George Mason University（计算机科学系，乔治·马歇尔大学）；Department of Information Sciences and Technology, George Mason University（信息科学与技术系，乔治·马歇尔大学）
Interpreting Context-Aware Human Preferences for Multi-Objective Robot Navigation
Hochschule Bonn-Rhein-Sieg, Germany（波恩-莱茵-锡格应用科学大学，德国）；University of Bonn, Germany（波恩大学，德国）；Lamarr Institute for Machine Learning and Artificial Intelligence, Germany（拉马尔机器学习与人工智能研究所，德国）
Navigating the Crowd: Non-linear MPC with Social Forces Dynamics for Human-Aware Robot Navigation
Department of Electronics and Telecommunications, Politecnico di Torino（电子与电信系，都灵理工大学）；School of Engineering, Pablo de Olavide University（工程学院，巴勃罗·德奥拉维德大学）
US Army Research Laboratory（美国陆军研究实验室）；Carnegie Mellon University（卡内基梅隆大学）
When working alongside human collaborators in dynamic and unstructured environments, such as disaster recovery or military operation, fast field adaptation is necessary for an unmanned ground vehicle (UGV) to perform its duties or learn novel tasks. In these scenarios, personnel and equipment are constrained, making training with minimal human supervision a desirable learning attribute. We address the problem of making UGVs more reliable and adaptable teammates with a novel framework that uses visual perception and inverse optimal control to learn traversal costs for environment features. Thro…
