Active perception systems maximizing information gain to support both monitoring and decision making have seen considerable application in recent work. In this paper, we propose and demonstrate a method of acquiring and extrapolating information in an active sensory system through use of a Bayesian Filter. Our approach is motivated by manufacturing processes, where automated visual tracking of system states may aid in fault diagnosis, certification of parts and safety; in extreme cases, our approach may enable novel manufacturing processes relying on monitoring solutions beyond passive perception. We demonstrate how using a Bayesian Filter in active perception scenarios permits reasoning about future actions based on measured as well as unmeasured but propagated state elements, thereby increasing substantially the quality of information available to decision making algorithms used in control of overarching processes. We demonstrate use of our active perception system in physical experiments, where we use a time-varying Kalman Filter to resolve uncertainty for a representative system capturing in additive manufacturing.
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Robotic perception models, such as Deep Neural Networks (DNNs), are becoming more computationally intensive and there are several models being trained with accuracy and latency trade-offs. However, modern latency accuracy trade-offs largely report mean accuracy for single-step vision tasks, but there is little work showing which model to invoke for multi-step control tasks in robotics. The key challenge in a multi-step decision making is to make use of the right models at right times to accomplish the given task. That is, the accomplishment of the task with a minimum control cost and minimum p…
Perceptual Factors for Environmental Modeling in Robotic Active Perception
Instituto de Investigación en Ingeniería de Aragón, Universidad de Zaragoza（萨拉戈萨大学阿拉贡工程研究院）；Institute of Geodesy and Geoinformation, University of Bonn（波恩大学大地测量与地理信息研究所）
Research Institutes of Sweden（瑞典研究机构）；University of California, Berkeley（加州大学伯克利分校）；Luleå University of Technology（吕勒奥理工大学）
This correspondence comments on the findings reported in a recent Science Robotics article by Mitrokhin et al. [1]. The main goal of this commentary is to expand on some of the issues touched on in that article. Our experience is that hyperdimensional computing is very different from other approaches to computation and that it can take considerable exposure to its concepts before attaining practically useful understanding. Therefore, in order to provide an overview of the area to the first time reader of [1], the commentary includes a brief historic overview as well as connects the findings of…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）；Cyberbotics（Cyberbotics公司）；Delft University of Technology（代尔夫特理工大学）；University of Freiburg（弗莱堡大学）；PAL Robotics（PAL机器人公司）；Tampere University（坦佩雷大学）；Agrointelli（Agrointelli公司）；Aarhus University（奥胡斯大学）
Existing Deep Learning (DL) frameworks typically do not provide ready-to-use solutions for robotics, where very specific learning, reasoning, and embodiment problems exist. Their relatively steep learning curve and the different methodologies employed by DL compared to traditional approaches, along with the high complexity of DL models, which often leads to the need of employing specialized hardware accelerators, further increase the effort and cost needed to employ DL models in robotics. Also, most of the existing DL methods follow a static inference paradigm, as inherited by the traditional …
University of Maryland（马里兰大学）；University of California, Berkeley（加州大学伯克利分校）；York University（约克大学）
Despite the recent successes in robotics, artificial intelligence and computer vision, a complete artificial agent necessarily must include active perception. A multitude of ideas and methods for how to accomplish this have already appeared in the past, their broader utility perhaps impeded by insufficient computational power or costly hardware. The history of these ideas, perhaps selective due to our perspectives, is presented with the goal of organizing the past literature and highlighting the seminal contributions. We argue that those contributions are as relevant today as they were decades…
Robotics has long been a field riddled with complex systems architectures whose modules and connections, whether traditional or learning-based, require significant human expertise and prior knowledge. Inspired by large pre-trained language models, this work introduces a paradigm for pre-training a general purpose representation that can serve as a starting point for multiple tasks on a given robot. We present the Perception-Action Causal Transformer (PACT), a generative transformer-based architecture that aims to build representations directly from robot data in a self-supervised fashion. Thro…
To better optimise the global food supply chain, robotic solutions are needed to automate tasks currently completed by humans. Namely, phenotyping, quality analysis and harvesting are all open problems in the field of agricultural robotics. Robotic perception is a key challenge for autonomous solutions to such problems as scene understanding and object detection are vital prerequisites to any grasping tasks that a robot may undertake. This work conducts a brief review of modern robot perception models and discusses their efficacy within the agri-food domain.
Learning to Optimize Edge Robotics: A Fast Integrated Perception-Motion-Communication Approach
Beijing University of Posts and Telecommunications（北京邮电大学）；South China University of Technology（华南理工大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Macau（澳门大学）
Robotics Research Center, IIIT Hyderabad（国际信息技术研究所海得拉巴分校机器人研究中心）；TCS Research（塔塔咨询服务公司研究部）
Table-top Rearrangement and Planning is a challenging problem that relies heavily on an excellent perception stack. The perception stack involves observing and registering the 3D scene on the table, detecting what objects are on the table, and how to manipulate them. Consequently, it greatly influences the system's task-planning and motion-planning stacks that follow. We present a comprehensive overview and discuss the different challenges associated with the perception module. This work is a result of our extensive involvement in the ICRA 2022 Open Cloud Robot Table Organization Challenge, in…
The ARC Centre of Excellence for Robotic Vision（澳大利亚研究理事会机器人视觉卓越中心）；The Centre for Robotics, Queensland University of Technology（昆士兰科技大学机器人中心）
As deep learning continues to dominate all state-of-the-art computer vision tasks, it is increasingly becoming an essential building block for robotic perception. This raises important questions concerning the safety and reliability of learning-based perception systems. There is an established field that studies safety certification and convergence guarantees of complex software systems at design-time. However, the unknown future deployment environments of an autonomous system and the complexity of learning-based perception make the generalization of design-time verification to run-time proble…
Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）
The integration of visual-tactile stimulus is common while humans performing daily tasks. In contrast, using unimodal visual or tactile perception limits the perceivable dimensionality of a subject. However, it remains a challenge to integrate the visual and tactile perception to facilitate robotic tasks. In this paper, we propose a novel framework for the cross-modal sensory data generation for visual and tactile perception. Taking texture perception as an example, we apply conditional generative adversarial networks to generate pseudo visual images or tactile outputs from data of the other m…
The following paper reviews recent developments in the field of optimization of space robotics. The extent of focus of this paper is on the perception (robotic sense of analyzing surroundings) in space robots in the exploration of extra-terrestrial planets. Robots play a crucial role in exploring extra-terrestrial and planetary bodies. Their advantages are far from being counted on finger tips. With the advent of autonomous robots in the field of robotics, the role for space exploration has further hustled up. Optimization of such autonomous robots has turned into a necessity of the hour. Opti…
The sense of touch plays a key role in enabling humans to understand and interact with surrounding environments. For robots, tactile sensing is also irreplaceable. While interacting with objects, tactile sensing provides useful information for the robot to understand the object, such as distributed pressure, temperature, vibrations and texture. During robot grasping, vision is often occluded by its end-effectors, whereas tactile sensing can measure areas that are not accessible by vision. In the past decades, a number of tactile sensors have been developed for robots and used for different rob…
A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University（机器人与自动信息系统研究所，人工智能学院，南开大学）；Tianjin Key Laboratory of Intelligent Robotics, Nankai University（智能机器人重点实验室，南开大学）；College of Artificial Intelligence, Nankai University（人工智能学院，南开大学）；Department of Computing, The Hong Kong Polytechnic University（computing 部门，香港理工大学）
Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference
College of Big Data and Internet, Shenzhen Technology University, China（大数据与互联网学院，深圳科技大学，中国）；Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, China（中德智能制造学院，深圳科技大学，中国）；Robotics Department, Institute of Smart Systems and Artificial Intelligence (ISSAI), Nazarbayev University, Kazakhstan（机器人系，智能系统与人工智能研究所（ISSAI），纳扎尔巴耶夫大学，哈萨克斯坦）；Foundation Robotics Labs, Germany（基础机器人实验室，德国）
While the most visible part of the safety verification process of automated vehicles concerns the planning and control system, it is often overlooked that safety of the latter crucially depends on the fault-tolerance of the preceding environment perception. Modern perception systems feature complex and often machine-learning-based components with various failure modes that can jeopardize the overall safety. At the same time, a verification by for example redundant execution is not always feasible due to resource constraints. In this paper, we address the need for feasible and efficient percept…
This paper describes the development of a humanoid robot called ARDOP. The goal of the project is to provide a modular, open-source, and inexpensive humanoid robot that would enable researchers to answer various problems related to robotic manipulation and perception. ARDOP primarily comprises of two functional units namely the perception and manipulation system, here we discuss the conceptualization and design methodology of these systems and proceed to present their performance results on simulation and various custom-designed experiments.
Donders Institute for Brain, Cognition and Behaviour（唐德斯脑、认知与行为研究所）；Radboud University（拉德堡德大学）
Unlike robots, humans learn, adapt and perceive their bodies by interacting with the world. Discovering how the brain represents the body and generates actions is of major importance for robotics and artificial intelligence. Here we discuss how neuroscience findings open up opportunities to improve current estimation and control algorithms in robotics. In particular, how active inference, a mathematical formulation of how the brain resists a natural tendency to disorder, provides a unified recipe to potentially solve some of the major challenges in robotics, such as adaptation, robustness, fle…
Accurate knowledge of object poses is crucial to successful robotic manipulation tasks, and yet most current approaches only work in laboratory settings. Noisy sensors and cluttered scenes interfere with accurate pose recognition, which is problematic especially when performing complex tasks involving object interactions. This is because most pose estimation algorithms focus only on estimating objects from a single frame, which means they lack continuity between frames. Further, they often do not consider resulting physical properties of the predicted scene such as intersecting objects or obje…
University of Nevada, Reno（内华达大学里诺分校）；University of Technology Sydney（悉尼科技大学）
This paper presents visual and 3D structure inspection for steel structures and bridges using a developed climbing robot. The robot can move freely on a steel surface, carry sensors, collect data and then send to the ground station in real time for monitoring as well as further processing. Steel surface image stitching and 3D map building are conducted to provide a current condition of the structure. Also, a computer vision-based method is implemented to detect surface defects on stitched images. The effectiveness of the climbing robot's inspection is tested in multiple circumstances to ensure…
Look-to-Touch: A Vision-Enhanced Proximity and Tactile Sensor for Distance and Geometry Perception in Robotic Manipulation
State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University（上海交通大学机械系统与振动国家重点实验室）；Robotics Institute, School of Mechanical Engineering, Shanghai Jiao Tong University（上海交通大学机械工程学院机器人研究所）
Collaborative Robotics and Intelligent Systems (CoRIS) Institute（协作机器人与智能系统（CoRIS）研究所）；Oregon State University（俄勒冈州立大学）
Rising global food demand and harsh working conditions make fruit harvest an important domain to automate. Peduncle localization is an important step for any automated fruit harvesting system, since fruit separation techniques are highly sensitive to peduncle location. Most work on peduncle localization has focused on computer vision, but peduncles can be difficult to visually access due to the cluttered nature of agricultural environments. Our work proposes an alternative method which relies on mechanical -- rather than visual -- perception to localize the peduncle. To estimate the location o…
Artificial Intelligence Research Institute (IIIA-CSIC)（人工智能研究所 (IIIA-CSIC)）；Computer Vision Center（计算机视觉中心）；Australian Centre for Field Robotics（澳大利亚野外机器人中心）；The University of Sydney（悉尼大学）
This paper addresses object perception applied to mobile robotics. Being able to perceive semantically meaningful objects in unstructured environments is a key capability in order to make robots suitable to perform high-level tasks in home environments. However, finding a solution for this task is daunting: it requires the ability to handle the variability in image formation in a moving camera with tight time constraints. The paper brings to attention some of the issues with applying three state of the art object recognition and detection methods in a mobile robotics scenario, and proposes met…
The development of automatic perception systems and techniques for bio-inspired flapping-wing robots is severely hampered by the high technical complexity of these platforms and the installation of onboard sensors and electronics. Besides, flapping-wing robot perception suffers from high vibration levels and abrupt movements during flight, which cause motion blur and strong changes in lighting conditions. This paper presents a perception dataset for bird-scale flapping-wing robots as a tool to help alleviate the aforementioned problems. The presented data include measurements from onboard sens…
Aerial drones have great potential to monitor large areas quickly and efficiently. Aquaculture is an industry that requires continuous water quality data to successfully grow and harvest fish. The Hybrid Aerial Underwater Robotic System (HAUCS) is designed to collect water quality data of aquaculture ponds to reduce labor costs for farmers. The routing of drones to cover each fish pond on an aquaculture farm can be reduced to the Vehicle Routing Problem. A dataset is created to simulate the distribution of ponds on a farm and is used to assess the HAUCS Path Planning Algorithm (HPP). Its perfo…
BMW Group（宝马集团）；University of Glasgow（格拉斯哥大学）；The Imperial College of Science, Technology and Medicine（帝国理工学院）；Donders Institute for Brain and Cognition, Radboud University（拉德堡德大学唐德斯脑认知研究所）
This work presents a novel active visuo-tactile based framework for robotic systems to accurately estimate pose of objects in dense cluttered environments. The scene representation is derived using a novel declutter graph (DG) which describes the relationship among objects in the scene for decluttering by leveraging semantic segmentation and grasp affordances networks. The graph formulation allows robots to efficiently declutter the workspace by autonomously selecting the next best object to remove and the optimal action (prehensile or non-prehensile) to perform. Furthermore, we propose a nove…
ShotcreteDepth: A Bi-modal Dataset for Robust Robotic Depth Perception in Shotcrete Construction Environments
Technical University of Denmark, Department of Electrical and Photonics Engineering（丹麦技术大学电气与光子工程系）；Pioneer Centre for AI（人工智能先锋中心）；Christiansen & Essenbæk A/S（Christiansen & Essenbæk 公司）
