As mobile robot capabilities improve and deployment times increase, tools to analyze the growing volume of data are becoming necessary. Current state-of-the-art logging, playback, and exploration systems are insufficient for practitioners seeking to discover systemic points of failure in robotic systems. This paper presents a suite of algorithms for similarity-based queries of robotic perception data and implements a system for storing 2D LiDAR data from many deployments cheaply and evaluating top-k queries for complete or partial scans efficiently. We generate compressed representations of laser scans via a convolutional variational autoencoder and store them in a database, where a light-weight dense network for distance function approximation is run at query time. Our query evaluator leverages the local continuity of the embedding space to generate evaluation orders that, in expectation, dominate full linear scans of the database. The accuracy, robustness, scalability, and efficiency of our system is tested on real-world data gathered from dozens of deployments and synthetic data generated by corrupting real data. We find our system accurately and efficiently identifies similar scans across a number of episodes where the robot encountered the same location, or similar indoor structures or objects.
Beyond Pixel Similarity: Task-Aware Evaluation of GAN-Based Synthetic Sonar Data for Robotic Perception
XITASO GmbH IT & Software Solutions（XITASO信息技术与软件解决方案有限公司）；National University of Sciences and Technology (NUST)（国立科技大学）；Technical University of Kaiserslautern（凯泽斯劳滕工业大学）
Artificial Intelligence Research Institute (IIIA-CSIC)（人工智能研究所 (IIIA-CSIC)）；Computer Vision Center（计算机视觉中心）；Australian Centre for Field Robotics（澳大利亚野外机器人中心）；The University of Sydney（悉尼大学）
This paper addresses object perception applied to mobile robotics. Being able to perceive semantically meaningful objects in unstructured environments is a key capability in order to make robots suitable to perform high-level tasks in home environments. However, finding a solution for this task is daunting: it requires the ability to handle the variability in image formation in a moving camera with tight time constraints. The paper brings to attention some of the issues with applying three state of the art object recognition and detection methods in a mobile robotics scenario, and proposes met…
European Centre for Soft Computing（欧洲软计算中心）；Universidad Politécnica de Madrid（马德里理工大学）
This work, inspired by the idea of "Computing with Words and Perceptions" proposed by Zadeh in 2001, focuses on how to transform measurements into perceptions for the problem of map building by Autonomous Mobile Robots. We propose to model the perceptions obtained from sonar-sensors as two grid maps: one for obstacles and another for empty spaces. The rules used to build and integrate these maps are expressed by linguistic descriptions and modeled by fuzzy rules. The main difference of this approach from other studies reported in the literature is that the method presented here is based on the…
University of Oxford（牛津大学）；Stanford University（斯坦福大学）
Determining the relative position and orientation of objects in an environment is a fundamental building block for a wide range of robotics applications. To accomplish this task efficiently in practical settings, a method must be fast, use common sensors, and generalize easily to new objects and environments. We present MSL-RAPTOR, a two-stage algorithm for tracking a rigid body with a monocular camera. The image is first processed by an efficient neural network-based front-end to detect new objects and track 2D bounding boxes between frames. The class label and bounding box is passed to the b…
Recent advances in learning-based perception systems have led to drastic improvements in the performance of robotic systems like autonomous vehicles and surgical robots. These perception systems, however, are hard to analyze and errors in them can propagate to cause catastrophic failures. In this paper, we consider the problem of synthesizing safe and robust controllers for robotic systems which rely on complex perception modules for feedback. We propose a counterexample-guided synthesis framework that iteratively builds simple surrogate models of the complex perception module and enables us t…
To better optimise the global food supply chain, robotic solutions are needed to automate tasks currently completed by humans. Namely, phenotyping, quality analysis and harvesting are all open problems in the field of agricultural robotics. Robotic perception is a key challenge for autonomous solutions to such problems as scene understanding and object detection are vital prerequisites to any grasping tasks that a robot may undertake. This work conducts a brief review of modern robot perception models and discusses their efficacy within the agri-food domain.
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
A vast literature shows that the learning-based visual perception model is sensitive to adversarial noises, but few works consider the robustness of robotic perception models under widely-existing camera motion perturbations. To this end, we study the robustness of the visual perception model under camera motion perturbations to investigate the influence of camera motion on robotic perception. Specifically, we propose a motion smoothing technique for arbitrary image classification models, whose robustness under camera motion perturbations could be certified. The proposed robustness certificati…
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Robotic perception models, such as Deep Neural Networks (DNNs), are becoming more computationally intensive and there are several models being trained with accuracy and latency trade-offs. However, modern latency accuracy trade-offs largely report mean accuracy for single-step vision tasks, but there is little work showing which model to invoke for multi-step control tasks in robotics. The key challenge in a multi-step decision making is to make use of the right models at right times to accomplish the given task. That is, the accomplishment of the task with a minimum control cost and minimum p…
Learning to Optimize Edge Robotics: A Fast Integrated Perception-Motion-Communication Approach
Beijing University of Posts and Telecommunications（北京邮电大学）；South China University of Technology（华南理工大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Macau（澳门大学）
Donders Institute for Brain, Cognition and Behaviour（唐德斯脑、认知与行为研究所）；Radboud University（拉德堡德大学）
Unlike robots, humans learn, adapt and perceive their bodies by interacting with the world. Discovering how the brain represents the body and generates actions is of major importance for robotics and artificial intelligence. Here we discuss how neuroscience findings open up opportunities to improve current estimation and control algorithms in robotics. In particular, how active inference, a mathematical formulation of how the brain resists a natural tendency to disorder, provides a unified recipe to potentially solve some of the major challenges in robotics, such as adaptation, robustness, fle…
Robotics Research Center, IIIT Hyderabad（国际信息技术研究所海得拉巴分校机器人研究中心）；TCS Research（塔塔咨询服务公司研究部）
Table-top Rearrangement and Planning is a challenging problem that relies heavily on an excellent perception stack. The perception stack involves observing and registering the 3D scene on the table, detecting what objects are on the table, and how to manipulate them. Consequently, it greatly influences the system's task-planning and motion-planning stacks that follow. We present a comprehensive overview and discuss the different challenges associated with the perception module. This work is a result of our extensive involvement in the ICRA 2022 Open Cloud Robot Table Organization Challenge, in…
The ARC Centre of Excellence for Robotic Vision（澳大利亚研究理事会机器人视觉卓越中心）；The Centre for Robotics, Queensland University of Technology（昆士兰科技大学机器人中心）
As deep learning continues to dominate all state-of-the-art computer vision tasks, it is increasingly becoming an essential building block for robotic perception. This raises important questions concerning the safety and reliability of learning-based perception systems. There is an established field that studies safety certification and convergence guarantees of complex software systems at design-time. However, the unknown future deployment environments of an autonomous system and the complexity of learning-based perception make the generalization of design-time verification to run-time proble…
Multi-modal perception for soft robotic interactions using generative models
The BioRobotics Institute, Sant’Anna School of Advanced Studies（圣安娜高级研究院生物机器人研究所）；Sant’Anna School of Advanced Studies（圣安娜高级研究院）；University College London（伦敦大学学院）
The vast majority of visual animals actively control their eyes, heads, and/or bodies to direct their gaze toward different parts of their environment. In contrast, recent applications of reinforcement learning in robotic manipulation employ cameras as passive sensors. These are carefully placed to view a scene from a fixed pose. Active perception allows animals to gather the most relevant information about the world and focus their computational resources where needed. It also enables them to view objects from different distances and viewpoints, providing a rich visual experience from which t…
Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）
The integration of visual-tactile stimulus is common while humans performing daily tasks. In contrast, using unimodal visual or tactile perception limits the perceivable dimensionality of a subject. However, it remains a challenge to integrate the visual and tactile perception to facilitate robotic tasks. In this paper, we propose a novel framework for the cross-modal sensory data generation for visual and tactile perception. Taking texture perception as an example, we apply conditional generative adversarial networks to generate pseudo visual images or tactile outputs from data of the other m…
The following paper reviews recent developments in the field of optimization of space robotics. The extent of focus of this paper is on the perception (robotic sense of analyzing surroundings) in space robots in the exploration of extra-terrestrial planets. Robots play a crucial role in exploring extra-terrestrial and planetary bodies. Their advantages are far from being counted on finger tips. With the advent of autonomous robots in the field of robotics, the role for space exploration has further hustled up. Optimization of such autonomous robots has turned into a necessity of the hour. Opti…
The sense of touch plays a key role in enabling humans to understand and interact with surrounding environments. For robots, tactile sensing is also irreplaceable. While interacting with objects, tactile sensing provides useful information for the robot to understand the object, such as distributed pressure, temperature, vibrations and texture. During robot grasping, vision is often occluded by its end-effectors, whereas tactile sensing can measure areas that are not accessible by vision. In the past decades, a number of tactile sensors have been developed for robots and used for different rob…
The development of automatic perception systems and techniques for bio-inspired flapping-wing robots is severely hampered by the high technical complexity of these platforms and the installation of onboard sensors and electronics. Besides, flapping-wing robot perception suffers from high vibration levels and abrupt movements during flight, which cause motion blur and strong changes in lighting conditions. This paper presents a perception dataset for bird-scale flapping-wing robots as a tool to help alleviate the aforementioned problems. The presented data include measurements from onboard sens…
Siemens（西门子）；California Institute of Technology（加州理工学院）
Active perception systems maximizing information gain to support both monitoring and decision making have seen considerable application in recent work. In this paper, we propose and demonstrate a method of acquiring and extrapolating information in an active sensory system through use of a Bayesian Filter. Our approach is motivated by manufacturing processes, where automated visual tracking of system states may aid in fault diagnosis, certification of parts and safety; in extreme cases, our approach may enable novel manufacturing processes relying on monitoring solutions beyond passive percept…
TRIAGE: Type-Routed Interventions via Aleatoric-Epistemic Gated Estimation in Robotic Manipulation and Adaptive Perception -- Don't Treat All Uncertainty the Same
University of Illinois at Chicago（伊利诺伊大学香槟分校）；Intel Labs（英特尔实验室）
NEC Corporation（日本电气股份有限公司）；Stanford University（斯坦福大学）
Today, even the most compute-and-power constrained robots can measure complex, high data-rate video and LIDAR sensory streams. Often, such robots, ranging from low-power drones to space and subterranean rovers, need to transmit high-bitrate sensory data to a remote compute server if they are uncertain or cannot scalably run complex perception or mapping tasks locally. However, today's representations for sensory data are mostly designed for human, not robotic, perception and thus often waste precious compute or wireless network resources to transmit unimportant parts of a scene that are unnece…
Cross-Modal Benchmarking for Robotic Perception in Natural Environments
CSIRO Robotics, CSIRO, Australia（CSIRO机器人研究所，CSIRO，澳大利亚）；University of Sydney (USyd), Australia（悉尼大学（USyd），澳大利亚）；Queensland University of Technology (QUT), Australia（昆士兰理工大学（QUT），澳大利亚）
A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University（机器人与自动信息系统研究所，人工智能学院，南开大学）；Tianjin Key Laboratory of Intelligent Robotics, Nankai University（智能机器人重点实验室，南开大学）；College of Artificial Intelligence, Nankai University（人工智能学院，南开大学）；Department of Computing, The Hong Kong Polytechnic University（computing 部门，香港理工大学）
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
We present an approach to continuous perception for robotic laundry tasks. Our assumption is that the visual prediction of a garment's shapes and weights is possible via a neural network that learns the dynamic changes of garments from video sequences. Continuous perception is leveraged during training by inputting consecutive frames, of which the network learns how a garment deforms. To evaluate our hypothesis, we captured a dataset of 40K RGB and 40K depth video sequences while a garment is being manipulated. We also conducted ablation studies to understand whether the neural network learns …
Unified Linear Parametric Map Modeling and Perception-aware Trajectory Planning for Mobile Robotics
School of Information Science and Engineering, Shenyang University of Technology（沈阳理工大学信息科学与工程学院）；State Key Laboratory of Synthetic Automation for Process Industries, Northeastern University（东北大学过程工业综合自动化重点实验室）；State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所机器人重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference
College of Big Data and Internet, Shenzhen Technology University, China（大数据与互联网学院，深圳科技大学，中国）；Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, China（中德智能制造学院，深圳科技大学，中国）；Robotics Department, Institute of Smart Systems and Artificial Intelligence (ISSAI), Nazarbayev University, Kazakhstan（机器人系，智能系统与人工智能研究所（ISSAI），纳扎尔巴耶夫大学，哈萨克斯坦）；Foundation Robotics Labs, Germany（基础机器人实验室，德国）
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）
While learning from synthetic training data has recently gained an increased attention, in real-world robotic applications, there are still performance deficiencies due to the so-called Sim-to-Real gap. In practice, this gap is hard to resolve with only synthetic data. Therefore, we focus on an efficient acquisition of real data within a Sim-to-Real learning pipeline. Concretely, we employ deep Bayesian active learning to minimize manual annotation efforts and devise an autonomous learning paradigm to select the data that is considered useful for the human expert to annotate. To achieve this, …
We present the design, implementation, and evaluation of RF-Grasp, a robotic system that can grasp fully-occluded objects in unknown and unstructured environments. Unlike prior systems that are constrained by the line-of-sight perception of vision and infrared sensors, RF-Grasp employs RF (Radio Frequency) perception to identify and locate target objects through occlusions, and perform efficient exploration and complex manipulation tasks in non-line-of-sight settings. RF-Grasp relies on an eye-in-hand camera and batteryless RFID tags attached to objects of interest. It introduces two main inno…
CYBRES GmbH（CYBRES有限公司）；Research Center of Advanced Robotics and Environmental Science（先进机器人与环境科学研究中心）
This work explores the application of nonlinear oscillators coupled by electric field in water for collective tasks in underwater robotics. Such coupled oscillators operate in clear and colloidal (mud, bottom silt) water and represent a collective electrochemical sensor that is sensitive to global environmental parameters, geometry of common electric field and spatial dynamics of autonomous underwater vehicles (AUVs). Implemented in hardware and software, this approach can be used to create global awareness in the group of robots, which possess limited sensing and communication capabilities. U…
This paper describes the development of a humanoid robot called ARDOP. The goal of the project is to provide a modular, open-source, and inexpensive humanoid robot that would enable researchers to answer various problems related to robotic manipulation and perception. ARDOP primarily comprises of two functional units namely the perception and manipulation system, here we discuss the conceptualization and design methodology of these systems and proceed to present their performance results on simulation and various custom-designed experiments.
Accurate knowledge of object poses is crucial to successful robotic manipulation tasks, and yet most current approaches only work in laboratory settings. Noisy sensors and cluttered scenes interfere with accurate pose recognition, which is problematic especially when performing complex tasks involving object interactions. This is because most pose estimation algorithms focus only on estimating objects from a single frame, which means they lack continuity between frames. Further, they often do not consider resulting physical properties of the predicted scene such as intersecting objects or obje…
To solve its task, a robot needs to have the ability to interpret its perceptions. In vision, this interpretation is particularly difficult and relies on the understanding of the structure of the scene, at least to the extent of its task and sensorimotor abilities. A robot with the ability to build and adapt this interpretation process according to its own tasks and capabilities would push away the limits of what robots can achieve in a non controlled environment. A solution is to provide the robot with processes to build such representations that are not specific to an environment or a situat…
