To achieve high-levels of autonomy, modern robots require the ability to detect and recover from anomalies and failures with minimal human supervision. Multi-modal sensor signals could provide more information for such anomaly detection tasks; however, the fusion of high-dimensional and heterogeneous sensor modalities remains a challenging problem. We propose a deep learning neural network: supervised variational autoencoder (SVAE), for failure identification in unstructured and uncertain environments. Our model leverages the representational power of VAE to extract robust features from high-dimensional inputs for supervised learning tasks. The training objective unifies the generative model and the discriminative model, thus making the learning a one-stage procedure. Our experiments on real field robot data demonstrate superior failure identification performance than baseline methods, and that our model learns interpretable representations. Videos of our results are available on our website: https://sites.google.com/illinois.edu/supervised-vae .
UnsOcc: 3D Semantic Occupancy Prediction in Unstructured Scene via Rendering Fusion
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Waytous Inc.（Waytous公司）
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of Science and Technology of China（中国科学技术大学）；Huazhong University of Science and Technology（华中科技大学）；Microsoft Research（微软研究院）
Towards Unified AI Drug Discovery with Multiple Knowledge Modalities
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）
Westview High School（韦斯特维尤高中）；Berkeley DeepDrive Center（伯克利深度驾驶中心）
Autonomous driving requires operation in different behavioral modes ranging from lane following and intersection crossing to turning and stopping. However, most existing deep learning approaches to autonomous driving do not consider the behavioral mode in the training strategy. This paper describes a technique for learning multiple distinct behavioral modes in a single deep neural network through the use of multi-modal multi-task learning. We study the effectiveness of this approach, denoted MultiNet, using self-driving model cars for driving in unstructured environments such as sidewalks and …
Evidence-Based Landing Site Selection and Vison-Based Landing for UAVs in Unstructured Environments
Aerospace Research Centre, National Research Council Canada（加拿大国家研究委员会航空航天研究中心）；Department of Mechanical and Industrial Engineering, Toronto Metropolitan University（多伦多 Metropolitan 大学机械与工业工程系）
University of Southern California（南加州大学）；Max-Planck-Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Imitation learning has traditionally been applied to learn a single task from demonstrations thereof. The requirement of structured and isolated demonstrations limits the scalability of imitation learning approaches as they are difficult to apply to real-world scenarios, where robots have to be able to execute a multitude of tasks. In this paper, we propose a multi-modal imitation learning framework that is able to segment and imitate skills from unlabelled and unstructured demonstrations by learning skill segmentation and imitation learning jointly. The extensive simulation results indicate t…
Robert Bosch GmbH（罗伯特·博世有限公司）；Ulm University（乌尔姆大学）；Karlsruher Institute of Technology（卡尔斯鲁厄理工学院）
This work presents a probabilistic deep neural network that combines LiDAR point clouds and RGB camera images for robust, accurate 3D object detection. We explicitly model uncertainties in the classification and regression tasks, and leverage uncertainties to train the fusion network via a sampling mechanism. We validate our method on three datasets with challenging real-world driving scenarios. Experimental results show that the predicted uncertainties reflect complex environmental uncertainty like difficulties of a human expert to label objects. The results also show that our method consiste…
University of the Witwatersrand（威特沃特斯兰德大学）；University of Johannesburg（约翰内斯堡大学）
Context-aware recommendation systems improve upon classical recommender systems by including, in the modelling, a user's behaviour. Research into context-aware recommendation systems has previously only considered the sequential ordering of items as contextual information. However, there is a wealth of unexploited additional multi-modal information available in auxiliary knowledge related to items. This study extends the existing research by evaluating a multi-modal recommendation system that exploits the inclusion of comprehensive auxiliary knowledge related to an item. The empirical results …
Future planetary missions will rely on rovers that can autonomously explore and navigate in unstructured environments. An essential element is the ability to recognize places that were already visited or mapped. In this work, we leverage the ability of stereo cameras to provide both visual and depth information, guiding the search and validation of loop closures from a multi-modal perspective. We propose to augment submaps that are created by aggregating stereo point clouds, with visual keyframes. Point clouds matches are found by comparing CSHOT descriptors and validated by clustering, while …
The S3LI Vulcano Dataset: A Dataset for Multi-Modal SLAM in Unstructured Planetary Environments
German Aerospace Center (DLR) Institute of Robotics and Mechatronics（德国航空航天中心（DLR）机器人与机电研究所）；KTH Royal Institute of Technology（皇家理工学院）；Deptartment of Intelligent Systems（智能系统部门）
Department of Engineering（工程系）；Aarhus University（奥胡斯大学）；Australian Centre for Field Robotics（澳大利亚田野机器人中心）；The University of Sydney（悉尼大学）
Reliable obstacle detection and classification in rough and unstructured terrain such as agricultural fields or orchards remains a challenging problem. These environments involve large variations in both geometry and appearance, challenging perception systems that rely on only a single sensor modality. Geometrically, tall grass, fallen leaves, or terrain roughness can mistakenly be perceived as nontraversable or might even obscure actual obstacles. Likewise, traversable grass or dirt roads and obstacles such as trees and bushes might be visually ambiguous. In this paper, we combine appearance-…
Colorado School of Mines（科罗拉多矿业学院）；DEVCOM Army Research Laboratory（DEVCOM陆军研究实验室）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
When robots operate in real-world off-road environments with unstructured terrains, the ability to adapt their navigational policy is critical for effective and safe navigation. However, off-road terrains introduce several challenges to robot navigation, including dynamic obstacles and terrain uncertainty, leading to inefficient traversal or navigation failures. To address these challenges, we introduce a novel approach for adaptation by negotiation that enables a ground robot to adjust its navigational behaviors through a negotiation process. Our approach first learns prediction models for va…
Multi-modal perception for soft robotic interactions using generative models
The BioRobotics Institute, Sant’Anna School of Advanced Studies（圣安娜高级研究院生物机器人研究所）；Sant’Anna School of Advanced Studies（圣安娜高级研究院）；University College London（伦敦大学学院）
Planning in unstructured environments is challenging -- it relies on sensing, perception, scene reconstruction, and reasoning about various uncertainties. We propose DeepSemanticHPPC, a novel uncertainty-aware hypothesis-based planner for unstructured environments. Our algorithmic pipeline consists of: a deep Bayesian neural network which segments surfaces with uncertainty estimates; a flexible point cloud scene representation; a next-best-view planner which minimizes the uncertainty of scene semantics using sparse visual measurements; and a hypothesis-based path planner that proposes multiple…
Colorado School of Mines（科罗拉多矿业学院）；US Army Research Laboratory（美国陆军研究实验室）
Terrain adaptation is an essential capability for a ground robot to effectively traverse unstructured off-road terrain in real-world field environments such as forests. However, the expected robot behaviors generated by terrain adaptation methods cannot always be executed accurately due to setbacks such as wheel slip and reduced tire pressure. To address this problem, we propose a novel approach for consistent behavior generation that enables the ground robot's actual behaviors to more accurately match expected behaviors while adapting to a variety of unstructured off-road terrain. Our approac…
OASIS: Real-Time Opti-Acoustic Sensing for Intervention Systems in Unstructured Environments
Applied Ocean Physics and Engineering, Woods Hole Oceanographic Institution, Deep Submergence Laboratory（应用海洋物理与工程，伍兹霍尔海洋研究所，深潜实验室）；Massachusetts Institute of Technology（麻省理工学院）
PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes
School of Automation and Intelligent Sensing（自动化与智能感知学院）；Shanghai Jiao Tong University（上海交通大学）；Global College（全球学院）；school of Computer Science（计算机科学学院）；Shanghai Key Laboratory of Navigation and Location Based Services（导航与位置基于服务上海市重点实验室）
Robotic Task Success Evaluation Under Multi-modal Non-Parametric Object Pose Uncertainty
University of Southern Denmark（南丹麦大学）；SDU Robotics（SDU机器人学（注：SDU为南丹麦大学缩写））；Mærsk Mc-Kinney Møller Institute (MMMI)（马士基麦金姆穆勒研究所）；Faculty of Engineering（工程学院）；Danish Institute for Advanced Studies (DIAS)（丹麦高级研究院）
Tsinghua University（清华大学）；Inner Mongolia University of Technology（内蒙古工业大学）
A deformable land-air robot is introduced with excellent driving and flying capabilities, offering a smooth switching mechanism between the two modes. An elaborate coupled dynamics model is established for the robot, including rotors, chassis, suspension, and the deformable structure. In addition, a model-based controller is designed for landing and mode switching in various unstructured conditions, such as slopes and curved surface. And considering locomotion and complex near-ground situations to achieve cooperation between the two fused modalities. This system was simulated in ADAMS/Simulink…
University of Coimbra（科英布拉大学）；University of Porto（波尔图大学）
More and more, new ways of interaction between humans and robots are desired, something that allow us to program a robot in an intuitive way, quickly and with a high-level of abstraction from the robot language. In this paper is presented a CAD-based system that allows users with basic skills in CAD and without skills in robot programming to generate robot programs from a CAD model of a robotic cell. When the CAD model reproduces exactly the real scenario, the system presents a satisfactory performance. On the contrary, when the CAD model does not reproduce exactly the real scenario or the cal…
The paper proposes a reliable and robust planning solution to the long range robotic navigation problem in extremely cluttered environments. A two-layer planning architecture is proposed that leverages both the environment map and the direct depth sensor information to ensure maximal information gain out of the onboard sensors. A frontier-based pose sampling technique is used with a fast marching cost-to-go calculation to select a goal pose and plan a path to maximize robot exploration rate. An artificial potential function approach, relying on direct depth measurements, enables the robot to f…
