This paper proposes an approach that predicts the road course from camera sensors leveraging deep learning techniques. Road pixels are identified by training a multi-scale convolutional neural network on a large number of full-scene-labeled night-time road images including adverse weather conditions. A framework is presented that applies the proposed approach to longer distance road course estimation, which is the basis for an augmented reality navigation application. In this framework long range sensor data (radar) and data from a map database are fused with short range sensor data (camera) to produce a precise longitudinal and lateral localization and road course estimation. The proposed approach reliably detects roads with and without lane markings and thus increases the robustness and availability of road course estimations and augmented reality navigation. Evaluations on an extensive set of high precision ground truth data taken from a differential GPS and an inertial measurement unit show that the proposed approach reaches state-of-the-art performance without the limitation of requiring existing lane markings.
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Army Research Laboratory（陆军研究实验室）
One of the key challenges in high speed off road navigation on ground vehicles is that the kinodynamics of the vehicle terrain interaction can differ dramatically depending on the terrain. Previous approaches to addressing this challenge have considered learning an inverse kinodynamics (IKD) model, conditioned on inertial information of the vehicle to sense the kinodynamic interactions. In this paper, we hypothesize that to enable accurate high-speed off-road navigation using a learned IKD model, in addition to inertial information from the past, one must also anticipate the kinodynamic intera…
Instituto de Investigación en Ingeniería de Aragón, (I3A). Universidad de Zaragoza（阿拉贡工程研究所（I3A），萨拉戈萨大学）
The visual functions of visual prostheses such as field of view, resolution and dynamic range, seriously restrict the person's ability to navigate in unknown environments. Implanted patients still require constant assistance for navigating from one location to another. Hence, there is a need for a system that is able to assist them safely during their journey. In this work, we propose an augmented reality navigation system for visual prosthesis that incorporates a software of reactive navigation and path planning which guides the subject through convenient, obstacle-free route. It consists on …
With a great amount of research going on in the field of autonomous vehicles or self-driving cars, there has been considerable progress in road detection and tracking algorithms. Most of these algorithms use GPS to handle road junctions and its subsequent decisions. However, there are places in the urban environment where it becomes difficult to get GPS fixes which render the junction decision handling erroneous or possibly risky. Vision-based junction detection, however, does not have such problems. This paper proposes a novel deep convolutional neural network architecture for disambiguation …
To help the blind people walk to the destination efficiently and safely in indoor environment, a novel wearable navigation device is presented in this paper. The locating, way-finding, route following and obstacle avoiding modules are the essential components in a navigation system, while it remains a challenging task to consider obstacle avoiding during route following, as the indoor environment is complex, changeable and possibly with dynamic objects. To address this issue, we propose a novel scheme which utilizes a dynamic sub-goal selecting strategy to guide the users to the destination an…
RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving
Jet Propulsion Laboratory (JPL)（喷气推进实验室）；California Institute of Technology (Caltech)（加州理工学院）；Swiss Federal Institute of Technology (ETH Zürich)（苏黎世联邦理工学院）
Tshwane University of Technology（茨瓦内理工大学）；Central University of Technology（中央理工大学）
Autonomous driving systems are broadly used equipment in the industries and in our daily lives, they assist in production, but are majorly used for exploration in dangerous or unfamiliar locations. Thus, for a successful exploration, navigation plays a significant role. Road detection is an essential factor that assists autonomous robots achieved perfect navigation. Various techniques using camera sensors have been proposed by numerous scholars with inspiring results, but their techniques are still vulnerable to these environmental noises: rain, snow, light intensity and shadow. In addressing …
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；General Motors Global R&D（通用汽车全球研发中心）
A safe and robust on-road navigation system is a crucial component of achieving fully automated vehicles. NVIDIA recently proposed an End-to-End algorithm that can directly learn steering commands from raw pixels of a front camera by using one convolutional neural network. In this paper, we leverage auxiliary information aside from raw images and design a novel network structure, called Auxiliary Task Network (ATN), to help boost the driving performance while maintaining the advantage of minimal training data and an End-to-End training method. In this network, we introduce human prior knowledg…
Research Institute on Embedded Electronic Systems (IRSEEM)（嵌入式电子系统研究所（IRSEEM））；ESIGELEC School of Engineering（ESIGELEC高等工程学院）；University of the Basque Country（巴斯克大学）；IKERBASQUE Basque Foundation for Science（IKERBASQUE巴斯克科学基金会）
In this paper, we propose a design for novel and experimental cloud computing systems. The proposed system aims at enhancing computational, communicational and annalistic capabilities of road navigation services by merging several independent technologies, namely vision-based embedded navigation systems, prominent Cloud Computing Systems (CCSs) and Vehicular Ad-hoc NETwork (VANET). This work presents our initial investigations by describing the design of a global generic system. The designed system has been experimented with various scenarios of video-based road services. Moreover, the associa…
Georgia Tech（佐治亚理工学院）；Facebook AI Research（Facebook 人工智能研究院）；Oregon State University（俄勒冈州立大学）
Recent work has presented embodied agents that can navigate to point-goal targets in novel indoor environments with near-perfect accuracy. However, these agents are equipped with idealized sensors for localization and take deterministic actions. This setting is practically sterile by comparison to the dirty reality of noisy sensors and actuations in the real world -- wheels can slip, motion sensors have error, actuations can rebound. In this work, we take a step towards this noisy reality, developing point-goal navigation agents that rely on visual estimates of egomotion under noisy action dyn…
SurgNavAR: An Augmented Reality Surgical Navigation Framework for Optical See-Through Head Mounted Displays
Biomedical Imaging Group Rotterdam, Department of Radiology & Nuclear Medicine, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心放射与核医学系鹿特丹生物医学成像组）；Department of Oral and Maxillofacial Surgery, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心口腔颌面外科）；University of Groningen（格罗宁根大学）；Department of High Performance Compute & Visualization, SURF bv（SURF bv高性能计算与可视化部）；Department of Neurosurgery, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心神经外科）；Trauma Research Unit, Department of Surgery, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心外科创伤研究单元）
The Australian National University（澳大利亚国立大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）；Yonsei University（延世大学）；Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Monash University（莫纳什大学）
Vision-and-Language Navigation (VLN) requires an agent to find a path to a remote location on the basis of natural-language instructions and a set of photo-realistic panoramas. Most existing methods take the words in the instructions and the discrete views of each panorama as the minimal unit of encoding. However, this requires a model to match different nouns (e.g., TV, table) against the same input view feature. In this work, we propose an object-informed sequential BERT to encode visual perceptions and linguistic instructions at the same fine-grained level, namely objects and words. Our seq…
GE Research（通用电气研究院）；Motional（Motional公司）；GE Digital（通用电气数字部门）
Off-road autonomous unmanned ground vehicles (UGVs) are being developed for military and commercial use to deliver crucial supplies in remote locations, help with mapping and surveillance, and to assist war-fighters in contested environments. Due to complexity of the off-road environments and variability in terrain, lighting conditions, diurnal and seasonal changes, the models used to perceive the environment must handle a lot of input variability. Current datasets used to train perception models for off-road autonomous navigation lack of diversity in seasons, locations, semantic classes, as w…
IIIT Hyderabad（印度海得拉巴国际信息技术学院）；Intel Bangalore（英特尔班加罗尔分公司）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
While several datasets for autonomous navigation have become available in recent years, they tend to focus on structured driving environments. This usually corresponds to well-delineated infrastructure such as lanes, a small number of well-defined categories for traffic participants, low variation in object or background appearance and strict adherence to traffic rules. We propose IDD, a novel dataset for road scene understanding in unstructured environments where the above assumptions are largely not satisfied. It consists of 10,004 images, finely annotated with 34 classes collected from 182 …
Establishing Reality-Virtuality Interconnections in Urban Digital Twins for Superior Intelligent Road Inspection and Simulation
College of Electronics & Information Engineering, Shanghai Research Institute for Intelligent Autonomous Systems, the State Key Laboratory of Intelligent Autonomous Systems, and Frontiers Science Center for Intelligent Autonomous Systems, Tongji University, Shanghai 201804, China（电子与信息工程学院，上海智能自主系统研究院，智能自主系统国家重点实验室，智能自主系统前沿科学中心，同济大学，上海）
Exploiting synthetic data to learn deep models has attracted increasing attention in recent years. However, the intrinsic domain difference between synthetic and real images usually causes a significant performance drop when applying the learned model to real world scenarios. This is mainly due to two reasons: 1) the model overfits to synthetic images, making the convolutional filters incompetent to extract informative representation for real images; 2) there is a distribution difference between synthetic and real data, which is also known as the domain adaptation problem. To this end, we prop…
Federal University of Santa Catarina（圣卡塔琳娜联邦大学）；National Institute for Digital Convergence（国家数字融合研究所）
One of the most relevant tasks in an intelligent vehicle navigation system is the detection of obstacles. It is important that a visual perception system for navigation purposes identifies obstacles, and it is also important that this system can extract essential information that may influence the vehicle's behavior, whether it will be generating an alert for a human driver or guide an autonomous vehicle in order to be able to make its driving decisions. In this paper we present an approach for the identification of obstacles and extraction of class, position, depth and motion information from…
