Autonomous navigation emerges from both motion and local visual perception in real-world environments. However, most successful robotic motion estimation methods (e.g. VO, SLAM, SfM) and vision systems (e.g. CNN, visual place recognition-VPR) are often separately used for mapping and localization tasks. Conversely, recent reinforcement learning (RL) based methods for visual navigation rely on the quality of GPS data reception, which may not be reliable when directly using it as ground truth across multiple, month-spaced traversals in large environments. In this paper, we propose a novel motion and visual perception approach, dubbed MVP, that unifies these two sensor modalities for large-scale, target-driven navigation tasks. Our MVP-based method can learn faster, and is more accurate and robust to both extreme environmental changes and poor GPS data than corresponding vision-only navigation methods. MVP temporally incorporates compact image representations, obtained using VPR, with optimized motion estimation data, including but not limited to those from VO or optimized radar odometry (RO), to efficiently learn self-supervised navigation policies via RL. We evaluate our method on two large real-world datasets, Oxford Robotcar and Nordland Railway, over a range of weather (e.g. overcast, night, snow, sun, rain, clouds) and seasonal (e.g. winter, spring, fall, summer) conditions using the new CityLearn framework; an interactive environment for efficiently training navigation agents. Our experimental results, on traversals of the Oxford RobotCar dataset with no GPS data, show that MVP can achieve 53% and 93% navigation success rate using VO and RO, respectively, compared to 7% for a vision-only method. We additionally report a trade-off between the RL success rate and the motion estimation precision.
Visual Computing and Computational Media Section (VCCM)（视觉计算与计算媒体部门）；College of Performance Visualization and Fine Arts (PVFA)（表演可视化与美术学院）；Department of Electrical and Computer Engineering (ECEN)（电气与计算机工程系）；Department of Computer Science and Engineering (CSCE)（计算机科学与工程系）
This work presents the most recent advances of the Robotic Testbed for Rendezvous and Optical Navigation (TRON) at Stanford University - the first robotic testbed capable of validating machine learning algorithms for spaceborne optical navigation. The TRON facility consists of two 6 degrees-of-freedom KUKA robot arms and a set of Vicon motion track cameras to reconfigure an arbitrary relative pose between a camera and a target mockup model. The facility includes multiple Earth albedo light boxes and a sun lamp to recreate the high-fidelity spaceborne illumination conditions. After the overview…
Imitation learning holds the promise to address challenging robotic tasks such as autonomous navigation. It however requires a human supervisor to oversee the training process and send correct control commands to robots without feedback, which is always prone to error and expensive. To minimize human involvement and avoid manual labeling of data in the robotic autonomous navigation with imitation learning, this paper proposes a novel semi-supervised imitation learning solution based on a multi-sensory design. This solution includes a suboptimal sensor policy based on sensor fusion to automatic…
Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots is logistically challenging, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic navigation that can leverage previously collected data …
Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing
Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute（威尔默眼科研究所）；Johns Hopkins Hospital（约翰斯·霍普金斯医院）；Technical University of Munich（慕尼黑工业大学）；Whiting School of Engineering（怀廷工程学院）
Rectilinear forms of snake-like robotic locomotion are anticipated to be an advantage in obstacle-strewn scenarios characterizing urban disaster zones, subterranean collapses, and other natural environments. The elongated, laterally-narrow footprint associated with these motion strategies is well-suited to traversal of confined spaces and narrow pathways. Navigation and path planning in the absence of global sensing, however, remains a pivotal challenge to be addressed prior to practical deployment of these robotic mechanisms. Several challenges related to visual processing and localization ne…
This work tackles scene understanding for outdoor robotic navigation, solely relying on images captured by an on-board camera. Conventional visual scene understanding interprets the environment based on specific descriptive categories. However, such a representation is not directly interpretable for decision-making and constrains robot operation to a specific domain. Thus, we propose to segment egocentric images directly in terms of how a robot can navigate in them, and tailor the learning problem to an autonomous navigation task. Building around an image segmentation network, we present a gen…
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong University of Science and Technology（香港科技大学）
Vision-Language Procedural Reasoning for Context-Aware Reward Modeling of Robotic Endovascular Guidewire Navigation
Department of Control Science and Engineering, College of Electronic and Information Engineering, and Shanghai Institute of Intelligent Science and Technology, Tongji University（同济大学电子与信息工程学院控制科学与工程系，以及上海智能科学与技术研究院）；Department of Electronic Engineering, Faculty of Engineering, The Chinese University of Hong Kong（香港中文大学工程学院电子工程系）；Shanghai Operation Robot Co., Ltd.（上海微创医疗机器人（集团）股份有限公司）
Active surgical robots lack acceptance in clinical practice, because they do not offer the flexibility and usability required for a versatile usage: the systems require a large installation space or a complicated registration step, where the preoperative plan is aligned to the patient and transformed to the base frame of the robot. In this paper, a navigation system for robotic osteotomies is designed, which uses the raw depth images from a camera mounted on the flange of a lightweight robot arm. Consequently, the system does not require any rigid attachment of the robot or fiducials to the bo…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；McGill University（麦吉尔大学）
We consider the task of underwater robot navigation for the purpose of collecting scientifically relevant video data for environmental monitoring. The majority of field robots that currently perform monitoring tasks in unstructured natural environments navigate via path-tracking a pre-specified sequence of waypoints. Although this navigation method is often necessary, it is limiting because the robot does not have a model of what the scientist deems to be relevant visual observations. Thus, the robot can neither visually search for particular types of objects, nor focus its attention on parts …
This paper reports on the development, execution, and open-sourcing of a new robotics course at MIT. The course is a modern take on "Visual Navigation for Autonomous Vehicles" (VNAV) and targets first-year graduate students and senior undergraduates with prior exposure to robotics. VNAV has the goal of preparing the students to perform research in robotics and vision-based navigation, with emphasis on drones and self-driving cars. The course spans the entire autonomous navigation pipeline; as such, it covers a broad set of topics, including geometric control and trajectory optimization, 2D and…
The Hong Kong University of Science and Technology（香港科技大学）
Visible light positioning (VLP) technology is a promising technique as it can provide high accuracy positioning based on the existing lighting infrastructure. However, existing approaches often require dense lighting distributions. Additionally, due to complicated indoor environments, it is still challenging to develop a robust VLP. In this work, we proposed loosely-coupled multi-sensor fusion method based on VLP and Simultaneous Localization and Mapping (SLAM), with light detection and ranging (LiDAR), odometry, and rolling shutter camera. Our method can provide accurate and robust robotics l…
HyperDCM: Dynamic Cluster Memory Replay in Hyperbolic Space for Continual Robotic Navigation Across Scenes
Software Engineering Institute, East China Normal University（华东师范大学软件工程学院）；School of Geospatial Information, Information Engineering University（信息工程大学地理空间信息学院）；University of Shanghai for Science and Technology（上海理工大学）；Shanghai Jiao Tong University（上海交通大学）
With growth in the use of autonomous Unmanned Ground Vehicle (UGV) for automated data collection from construction projects, the problem of inter-disciplinary semantic data sharing and exchanges between construction and robotic domains has attracted construction stakeholders' attention. Cross-domain data translation requires detailed specifications especially when it comes to semantic data translation. Building Information Modeling (BIM) and Geographic Information System (GIS) are the two technologies to capture and store construction data for indoor structure and outdoor environment respectiv…
Simon Fraser University（西蒙弗雷泽大学）；School of Computing Science（计算机科学学院）；Burnaby, BC Canada（加拿大不列颠哥伦比亚省本拿比）
This article describes a dataset collected in a set of experiments that involves human participants and a robot. The set of experiments was conducted in the computing science robotics lab in Simon Fraser University, Burnaby, BC, Canada, and its aim is to gather data containing common gestures, movements, and other behaviours that may indicate humans' navigational intent relevant for autonomous robot navigation. The experiment simulates a shopping scenario where human participants come in to pick up items from his/her shopping list and interact with a Pepper robot that is programmed to help the…
National University of Sciences and Technology（巴基斯坦国立科技大学）
Robots have revolutionized our way of life in recent years.One of the domains that has not yet completely benefited from the robotic automation is the agricultural sector. Agricultural Robotics should complement humans in the arduous tasks during different sub-domains of this sector. Extensive research in Agricultural Robotics has been carried out in Japan, USA, Australia and Germany focusing mainly on the heavy agricultural machinery. Pakistan is an agricultural rich country and its economy and food security are closely tied with agriculture in general and wheat in particular. However, agricu…
This paper presents an implementation of autonomous navigation functionality based on Robot Operating System (ROS) on a wheeled differential drive mobile platform called Eddie robot. ROS is a framework that contains many reusable software stacks as well as visualization and debugging tools that provides an ideal environment for any robotic project development. The main contribution of this paper is the description of the customized hardware and software system setup of Eddie robot to work with an autonomous navigation system in ROS called Navigation Stack and to implement one application use c…
Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Amsterdam（阿姆斯特丹大学）
Solving robotic navigation tasks via reinforcement learning (RL) is challenging due to their sparse reward and long decision horizon nature. However, in many navigation tasks, high-level (HL) task representations, like a rough floor plan, are available. Previous work has demonstrated efficient learning by hierarchal approaches consisting of path planning in the HL representation and using sub-goals derived from the plan to guide the RL policy in the source task. However, these approaches usually neglect the complex dynamics and sub-optimal sub-goal-reaching capabilities of the robot during pla…
Reinforcement Learning for Follow-the-Leader Robotic Endoscopic Navigation via Synthetic Data
School of Computer Science and Engineering, The University of New South Wales（计算机科学与工程学院，新南威尔士大学）；School of Mechanical and Manufacturing Engineering, The University of New South Wales（机械与制造工程学院，新南威尔士大学）
RoboRAN: A Unified Robotics Framework for Reinforcement Learning-Based Autonomous Navigation
Space Robotics Research Group, SnT University of Luxembourg（锡尔特大学卢森堡大学空间机器人研究组）；Georgia Tech-Europe IRL2958（佐治亚理工学院欧洲分校IRL2958）；GT-CNRS, Metz, France（GT-CNRS，法国梅茨）；Earth Species Project（地球物种项目）
Flocking model has been widely used to control robotic swarm. However, with the increasing scalability, there exist complex conflicts for robotic swarm in autonomous navigation, brought by internal pattern maintenance, external environment changes, and target area orientation, which results in poor stability and adaptability. Hence, optimizing the flocking model for robotic swarm in autonomous navigation is an important and meaningful research domain.
History-Conditioned Spatio-Temporal Visual Token Pruning for Efficient Vision-Language Navigation
Department of Computer and Information Sciences at the University of Delaware（德克萨斯大学达勒姆分校计算机与信息科学系）；University of Maryland’s Department of Computer Science（马里兰大学计算机科学系）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
IntelliMove: Enhancing Robotic Planning with Semantic Mapping
S.A.S. euroDAO（S.A.S. 欧元道）；LIRMM（蒙彼利埃计算机科学、机器人学、微电子与微电子实验室）；CNRS（法国国家科学研究中心）；Université de Montpellier（蒙彼利埃大学）；Université de Picardie Jules Verne（儒勒·凡尔纳皮卡第大学）
The Hong Kong Polytechnic University（香港理工大学）；Research Institute for Smart Ageing, PolyU（香港理工大学智衰研究所）
The scoliosis progression in adolescents requires close monitoring to timely take treatment measures. Ultrasound imaging is a radiation-free and low-cost alternative in scoliosis assessment to X-rays, which are typically used in clinical practice. However, ultrasound images are prone to speckle noises, making it challenging for sonographers to detect bony features and follow the spine's curvature. This paper introduces a robotic-ultrasound approach for spinal curvature tracking and automatic navigation. A fully connected network with deconvolutional heads is developed to locate the spinous pro…
Chinese University of Hong Kong（香港中文大学）；Southern University of Science and Technology（南方科技大学）；Shenzhen Research Institute of the Chinese University of Hong Kong（香港中文大学深圳研究院）
In recent years, simultaneous magnetic actuation and localization (SMAL) for active wireless capsule endoscopy (WCE) has been intensively studied to improve the efficiency and accuracy of the examination. In this paper, we propose an autonomous magnetic navigation framework for active WCE that mimics the "insertion" and "withdrawal" procedures performed by an expert physician in conventional colonoscopy, thereby enabling efficient and accurate navigation of a robotic capsule endoscope in the intestine with minimal user effort. First, the capsule is automatically propelled through the unknown i…
Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments. Inspired by the robotics community, we propose a modular approach to VLN using topological maps. Given a natural language instruction and topological map, our approach leverages attention mechanisms to predict a navigation plan in the map. The plan is then executed with low-level actions (e.g. forward, rotate) using a robust controller. Experiments show that our method outperforms previous end-to-end approaches, generates interpretable navigati…
Social navigation research is performed on a variety of robotic platforms, scenarios, and environments. Making comparisons between navigation algorithms is challenging because of the effort involved in building these systems and the diversity of platforms used by the community; nonetheless, evaluation is critical to understanding progress in the field. In a step towards reproducible evaluation of social navigation algorithms, we propose the Social Environment for Autonomous Navigation (SEAN). SEAN is a high visual fidelity, open source, and extensible social navigation simulation platform whic…
In the last decade, autonomous navigation for roboticshas been leveraged by deep learning and other approachesbased on machine learning. These approaches have demon-strated significant advantages in robotics performance. Butthey have the disadvantage that they require a lot of data toinfer knowledge. In this paper, we present an algorithm forbuilding 2D maps with attributes that make them useful fortraining and testing machine-learning-based approaches.The maps are based on dungeons environments where sev-eral random rooms are built and then those rooms are con-nected. In addition, we provide …
