Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Tasks
Purdue University（普渡大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Maryland, College Park（马里兰大学帕克分校）；University of Central Florida（中佛罗里达大学）
While LLMs excel in processing text in these human conversations, they struggle with the nuances of verbal instructions in scenarios like social navigation, where ambiguity and uncertainty can erode trust in robotic and other AI systems. We can address this shortcoming by moving beyond text and additionally focusing on the paralinguistic features of these audio responses. These features are the aspects of spoken communication that do not involve the literal wording (lexical content) but convey meaning and nuance through how something is said. We present Beyond Text: an approach that improves LLM decision-making by integrating audio transcription along with a subsection of these features, which focus on the affect and more relevant in human-robot conversations.This approach not only achieves a 70.26% winning rate, outperforming existing LLMs by 22.16% to 48.30% (gemini-1.5-pro and gpt-3.5 respectively), but also enhances robustness against token manipulation adversarial attacks, highlighted by a 22.44% less decrease ratio than the text-only language model in winning rate. Beyond Text' marks an advancement in social robot navigation and broader Human-Robot interactions, seamlessly integrating text-based guidance with human-audio-informed language models.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Semantic cues and statistical regularities in real-world environment layouts can improve efficiency for navigation in novel environments. This paper learns and leverages such semantic cues for navigating to objects of interest in novel environments, by simply watching YouTube videos. This is challenging because YouTube videos don't come with labels for actions or goals, and may not even showcase optimal behavior. Our method tackles these challenges through the use of Q-learning on pseudo-labeled transition quadruples (image, action, next image, reward). We show that such off-policy Q-learning …
Jacobs University Bremen（不来梅雅各布大学）；ShanghaiTech University（上海科技大学）
Underwater robot interventions require a high level of safety and reliability. A major challenge to address is a robust and accurate acquisition of localization estimates, as it is a prerequisite to enable more complex tasks, e.g. floating manipulation and mapping. State-of-the-art navigation in commercial operations, such as oil & gas production (OGP), rely on costly instrumentation. These can be partially replaced or assisted by visual navigation methods, especially in deep-sea scenarios where equipment deployment has high costs and risks. Our work presents a multimodal approach that adapts …
The gap between simulation and the real-world restrains many machine learning breakthroughs in computer vision and reinforcement learning from being applicable in the real world. In this work, we tackle this gap for the specific case of camera-based navigation, formulating it as following a visual cue in the foreground with arbitrary backgrounds. The visual cue in the foreground can often be simulated realistically, such as a line, gate or cone. The challenge then lies in coping with the unknown backgrounds and integrating both. As such, the goal is to train a visual agent on data captured in …
Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human languages facilitate communication by instructions in natural languages. Nevertheless, a plethora of other forms of communication is left unexplored. In fact, human communication originated in gestures and oftentimes is delivered through multimodal cues, e.g…
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学）；INRIA（法国国家信息与自动化研究所）
This paper addresses the problem of building augmented metric representations of scenes with semantic information from RGB-D images. We propose a complete framework to create an enhanced map representation of the environment with object-level information to be used in several applications such as human-robot interaction, assistive robotics, visual navigation, or in manipulation tasks. Our formulation leverages a CNN-based object detector (Yolo) with a 3D model-based segmentation technique to perform instance semantic segmentation, and to localize, identify, and track different classes of objec…
Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Department of Computer Science, Emory University（埃默里大学计算机科学系）；Department of Computer Science, University of Wisconsin-Madison（威斯康星大学麦迪逊分校计算机科学系）；Tsinghua University（清华大学）；Innovation and Research and Development Department, BoardWare Information System Company（BoardWare信息系统公司创新与研发部）；School of Mathematics, Peking University（北京大学数学学院）
Berkeley AI Research (BAIR)（伯克利人工智能研究院（BAIR））；University of California, Berkeley（加州大学伯克利分校）
A general-purpose intelligent robot must be able to learn autonomously and be able to accomplish multiple tasks in order to be deployed in the real world. However, standard reinforcement learning approaches learn separate task-specific policies and assume the reward function for each task is known a priori. We propose a framework that learns event cues from off-policy data, and can flexibly combine these event cues at test time to accomplish different tasks. These event cue labels are not assumed to be known a priori, but are instead labeled using learned models, such as computer vision detect…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
A robot guide dog has compelling advantages over animal guide dogs for its cost-effectiveness, potential for mass production, and low maintenance burden. However, despite the long history of guide dog robot research, previous studies were conducted with little or no consideration of how the guide dog handler and the guide dog work as a team for navigation. To develop a robotic guiding system that is genuinely beneficial to blind or visually impaired individuals, we performed qualitative research, including interviews with guide dog handlers and trainers and first-hand blindfold walking experie…
Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments
Department of Computer Science, George Mason University（计算机科学系，乔治·马歇尔大学）；Department of Information Sciences and Technology, George Mason University（信息科学与技术系，乔治·马歇尔大学）
In this paper, we will review the process of evidence accumulation in the PSEIKI system for expectation-driven interpretation of images of 3-D scenes. Expectations are presented to PSEIKI as a geometrical hierarchy of abstractions. PSEIKI's job is then to construct abstraction hierarchies in the perceived image taking cues from the abstraction hierarchies in the expectations. The Dempster-Shafer formalism is used for associating belief values with the different possible labels for the constructed abstractions in the perceived image. This system has been used successfully for autonomous navigat…
Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学联合具身智能研究实验室）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语种认知计算联合国际研究实验室）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
University of Hamburg（汉堡大学）；SoftBank Robotics Europe（软银机器人欧洲公司）
Service robots need to show appropriate social behaviour in order to be deployed in social environments such as healthcare, education, retail, etc. Some of the main capabilities that robots should have are navigation and conversational skills. If the person is impatient, the person might want a robot to navigate faster and vice versa. Linguistic features that indicate politeness can provide social cues about a person's patient and impatient behaviour. The novelty presented in this paper is to dynamically incorporate politeness in robotic dialogue systems for navigation. Understanding the polit…
Cog-GA: A Large Language Models-based Generative Agent for Vision-Language Navigation in Continuous Environments
Institute of Automation, Chinese Academy of Science (CASIA)（中国科学院自动化研究所）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；The University of Hong Kong（香港大学）
SONAR: Semantic-Object Navigation with Aggregated Reasoning through a Cross-Modal Inference Paradigm
Shenzhen Key Laboratory of Robotics Perception and Intelligence（机器人感知与智能深圳重点实验室）；Department of Electronic and Electrical Engineering（电子与电气工程系）；Southern University of Science and Technology（南方科技大学）；Research Institute of Multiple Agents and Embodied Intelligence（多智能体与具身智能研究院）；Peng Cheng Laboratory（鹏城实验室）；Robotics and Microsystems Center（机器人与微系统中心）；School of Mechanical and Electric Engineering（机械与电子工程学院）；Soochow University（苏州大学）；Xiamen Key Laboratory of Visual Perception Technology and Application（视觉感知技术与应用厦门重点实验室）；Reconova Information Technology Co., Ltd.（Reconova信息技术有限公司）
UT Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook人工智能研究院）
Recent work on audio-visual navigation assumes a constantly-sounding target and restricts the role of audio to signaling the target's position. We introduce semantic audio-visual navigation, where objects in the environment make sounds consistent with their semantic meaning (e.g., toilet flushing, door creaking) and acoustic events are sporadic or short in duration. We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target. Our model's persistent multimodal memory e…
Learning Robot Visual Navigation in Crowds via Intention-Aware Scene Representations
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, SUSTech（南方科技大学电子与电气工程系深圳市机器人感知与智能重点实验室）；Jiaxing Research Institute, SUSTech（南方科技大学嘉兴研究院）；Research Institute of MA&EI, Peng Cheng Laboratory（鹏城实验室机器人与人工智能研究所）
What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
University of Technology Nuremberg（纽伦堡工业大学）；ETH Zürich（苏黎世联邦理工学院）；Inria, Ecole Normale Supérieure, CNRS, PSL Research University（法国国家信息与自动化研究所，巴黎高等师范学校，法国国家科学研究中心，巴黎文理研究大学）
University of Freiburg（弗赖堡大学）；Google Research（谷歌研究院）；University of Technology Nuremberg（纽伦堡工业大学）
While interacting in the world is a multi-sensory experience, many robots continue to predominantly rely on visual perception to map and navigate in their environments. In this work, we propose Audio-Visual-Language Maps (AVLMaps), a unified 3D spatial map representation for storing cross-modal information from audio, visual, and language cues. AVLMaps integrate the open-vocabulary capabilities of multimodal foundation models pre-trained on Internet-scale data by fusing their features into a centralized 3D voxel grid. In the context of navigation, we show that AVLMaps enable robot systems to i…
Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control
Graduate School of Information Science and Technology, Hokkaido University（弘前大学信息科学与技术研究生院）；Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）
Robust Dynamic Object Detection in Cluttered Indoor Scenes via Learned Spatiotemporal Cues
Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente（代尔夫特理工大学电子工程、数学与计算机科学学院）；Department of Electrical Engineering, Wright State University（怀特塞德大学电子工程系）；Massachusetts Institute of Technology（麻省理工学院）
Politecnico di Torino（都灵理工大学）；Boston University（波士顿大学）
Drawing inspiration from biology, we describe the way in which visual sensing with a monocular camera can provide a reliable signal for navigation of mobile robots. The work takes inspiration from a classic paper by Lee and Reddish (Nature, 1981, https://doi.org/10.1038/293293a0) in which they outline a behavioral strategy pursued by diving sea birds based on a visual cue called time-to-contact. A closely related concept of time-to-transit, tau, is defined, and it is shown that idealized steering laws based on monocular camera perceptions of tau can reliably and robustly steer a mobile vehicle…
LERR-Nav: Learning to Coordinate Exploration, Recovery, and Reminiscing for Object Navigation
Hong Kong Polytechnic University（香港理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；DeepRobotics（深度机器人公司）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangzhou University（广州大学）
Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation
City University of Hong Kong, Dongguan Campus（香港城市大学东莞校区）；The University of Melbourne（墨尔本大学）；Tsinghua University（清华大学）；Baidu Inc.（百度公司）；University of Chinese Academy of Science（中国科学院大学）；National University of Singapore（新加坡国立大学）
University of Bonn（波恩大学）；Robert Bosch GmbH（罗伯特·博世有限公司）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）
We investigate the task of object goal navigation in unknown environments where the target is specified by a semantic label (e.g. find a couch). Such a navigation task is especially challenging as it requires understanding of semantic context in diverse settings. Most of the prior work tackles this problem under the assumption of a discrete action policy whereas we present an approach with continuous control which brings it closer to real world applications. We propose a deep neural network architecture and loss function to predict dense cost maps that implicitly contain semantic context and g…
In this paper, we present an algorithm to efficiently learn socially-compliant navigation policies from observations of human trajectories. As mobile robots come to inhabit and traffic social spaces, they must account for social cues and behave in a socially compliant manner. We focus on learning such cues from examples. We describe an inverse reinforcement learning based algorithm which learns from human trajectory observations without knowing their specific actions. We increase the sample-efficiency of our approach over alternative methods by leveraging the notion of a replay buffer (found i…
The Sword, Shield, and Achilles' Heel: Characterizing the Linguistic Inductive Bias of Large Language Models for Spatial Reasoning in Navigation Planning
East China Normal University（东华大学）；Information Engineering University（信息工程大学）；Zhengzhou University（郑州大学）
Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation
Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center（新疆多模态智能处理与信息安全工程技术创新中心）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；School of Electrical Engineering and Automation, Tianjin University of Technology（天津理工大学电气工程与自动化学院）
Spatial-Aware Conditioned Fusion for Audio-Visual Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学具身智能联合研究实验室）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语种认知计算联合国际研究实验室）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
