Go Fetch: Mobile Manipulation in Unstructured Environments
School of Information Technologies, University of Sydney（悉尼大学信息技术学院）；ETH Zürich（苏黎世联邦理工学院）
Kenneth Blomqvist, Michel Breyer, Andrei Cramariuc, Julian Förster, Margarita Grinvald, Florian Tschopp, Jen Jen Chung, Lionel Ott, Juan Nieto, Roland Siegwart
备注：
Kenneth Blomqvist, Michel Breyer, Andrei Cramariuc, Julian Förster, Margarita Grinvald, and Florian Tschopp contributed equally to this work
发表信息：
ICRA 2020 Workshop on Perception, Action, Learning: From Metric-Semantic Scene Understanding to High-level Task Execution, 2020
With humankind facing new and increasingly large-scale challenges in the medical and domestic spheres, automation of the service sector carries a tremendous potential for improved efficiency, quality, and safety of operations. Mobile robotics can offer solutions with a high degree of mobility and dexterity, however these complex systems require a multitude of heterogeneous components to be carefully integrated into one consistent framework. This work presents a mobile manipulation system that combines perception, localization, navigation, motion planning and grasping skills into one common workflow for fetch and carry applications in unstructured indoor environments. The tight integration across the various modules is experimentally demonstrated on the task of finding a commonly available object in an office environment, grasping it, and delivering it to a desired drop-off location. The accompanying video is available at https://youtu.be/e89_Xg1sLnY.
Indian Institute of Technology, Kanpur（印度理工学院坎普尔分校）
A robotic solution for the unmanned ground vehicles (UGVs) to execute the highly complex task of object manipulation in an autonomous mode is presented. This paper primarily focuses on developing an autonomous robotic system capable of assembling elementary blocks to build the large 3D structures in GPS-denied environments. The key contributions of this system paper are i) Designing of a deep learning-based unified multi-task visual perception system for object detection, part-detection, instance segmentation, and tracking, ii) an electromagnetic gripper design for robust grasping, and iii) sy…
Temporally Consistent Unsupervised Segmentation for Mobile Robot Perception
Oden Institute for Computational Engineering & Sciences, University of Texas at Austin（德纳学院，德克萨斯大学奥斯汀分校）；DEVCOM Army Research Laboratory, Adelphi, MD, United States（陆军研究实验室，阿德菲，马里兰州，美国）；Department of Electrical and Computer Engineering, University of Massachusetts Dartmouth（电气与计算机工程系，马萨诸塞大学达特茅斯分校）
Perceptive Hierarchical-Task MPC for Sequential Mobile Manipulation in Unstructured Semi-Static Environments
University of Toronto Institute for Aerospace Studies (UTIAS) and the Vector Institute for Artificial Intelligence（多伦多大学航空航天研究所（UTIAS）和人工智能向量研究所）；Learning Systems and Robotics lab at the Technical University of Munich and the Munich Institute for Robotics and Machine Intelligence (MIRMI)（慕尼黑技术大学学习系统与机器人实验室和慕尼黑机器人与机器智能研究所（MIRMI））；School of Computing Science, Faculty of Applied Sciences, Simon Fraser University（西蒙弗雷泽大学应用科学学院计算机科学学院）
The FetchIt! Mobile Manipulation Challenge, held at the IEEE International Conference on Robots and Automation (ICRA) in May 2019, offered an environment with complex and integrated task sets, irregular objects, confined space, and machining, introducing new challenges in the mobile manipulation domain. Here we describe our efforts to address these challenges by demonstrating the assembly of a kit of mechanical parts in a caddy. In addition to implementation details, we examine the issues in this task set extensively, and we discuss our software architecture in the hope of providing a base for…
Robots are being increasingly used in the fight against highly-infectious diseases such as Ebola, MERS, and SARS-COV-2. Many of the robots that are being used employ ultraviolet lights mounted on a mobile base to inactivate the pathogens. However, these lights are often mounted in a fixed configuration and do not provide adequate decontamination of horizontal surfaces, which can be a major source of cross-contamination. In the paper, we describe the design, implementation, and testing of an Ultraviolet Germicidal Irradiation (UVGI) system implemented on a mobile manipulation robot. A human sup…
University of Freiburg（弗莱堡大学）；University of Bremen（不来梅大学）
Service robots in the future need to execute abstract instructions such as "fetch the milk from the fridge". To translate such instructions into actionable plans, robots require in-depth background knowledge. With regards to interactions with doors and drawers, robots require articulation models that they can use for state estimation and motion planning. Existing frameworks model articulated connections as abstract concepts such as prismatic, or revolute, but do not provide a parameterized model of these connections for computation. In this paper, we introduce a novel framework that uses symbo…
M4Diffuser: Multi-View Diffusion Policy with Manipulability-Aware Control for Robust Mobile Manipulation
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg（汉堡大学信息学院技术多模态系统部门）；Technical University of Munich（慕尼黑技术大学）；Agile Robots SE（敏捷机器人有限公司）
Scalable Trajectory Generation for Whole-Body Mobile Manipulation
Institute for AI, Peking University（人工智能研究院，北京大学）；Institute of Unmanned System, Beihang University（无人系统研究院，北航）；School of Psychological and Cognitive Sciences, Peking University（心理学与认知科学学院，北京大学）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京行为与心理健康重点实验室，北京大学）；Yuanpei College, Peking University（元培学院，北京大学）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，PKU-武汉人工智能研究所）
Stereo Multistage Spatial Attention for Real-Time Mobile Manipulation Under Visual Scale Variation and Disturbances
Department of Intermedia Art and Science, Waseda University（媒体艺术与科学系，早稻田大学）；SB Intuitions Corp.（SB Intuitions公司）；Research and Development Group, Hitachi, Ltd.（日立株式会社研发部）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Heriot-Watt University（赫瑞-瓦特大学）
Agile control of mobile manipulator is challenging because of the high complexity coupled by the robotic system and the unstructured working environment. Tracking and grasping a dynamic object with a random trajectory is even harder. In this paper, a multi-task reinforcement learning-based mobile manipulation control framework is proposed to achieve general dynamic object tracking and grasping. Several basic types of dynamic trajectories are chosen as the task training set. To improve the policy generalization in practice, random noise and dynamics randomization are introduced during the train…
As autonomous service robots become more affordable and thus available also for the general public, there is a growing need for user friendly interfaces to control the robotic system. Currently available control modalities typically expect users to be able to express their desire through either touch, speech or gesture commands. While this requirement is fulfilled for the majority of users, paralyzed users may not be able to use such systems. In this paper, we present a novel framework, that allows these users to interact with a robotic service assistant in a closed-loop fashion, using only th…
Seconda Università degli Studi di Napoli（那不勒斯第二大学）
A reactive obstacle avoidance method for mobile manipulators is presented. The objectives of the developed algorithm are twofold. The first one is to find a trajectory in the configuration space of a mobile manipulator so as to follow a given trajectory in the task space. The second objective consists in locally adjusting the trajectory in the configuration space in order to avoid collisions with potentially moving obstacles and self-collisions in unstructured and dynamic environments. The perception is exclusively based on a set of proximity sensors distributed on the robot mechanical structu…
University of Michigan（密歇根大学）；Morehouse College（莫尔豪斯学院）
Articulated objects pose a unique challenge for robotic perception and manipulation. Their increased number of degrees-of-freedom makes tasks such as localization computationally difficult, while also making the process of real-world dataset collection unscalable. With the aim of addressing these scalability issues, we propose Neural Articulated Radiance Fields (NARF22), a pipeline which uses a fully-differentiable, configuration-parameterized Neural Radiance Field (NeRF) as a means of providing high quality renderings of articulated objects. NARF22 requires no explicit knowledge of the object…
Risk-Aware Reinforcement Learning for Mobile Manipulation
Oxford Robotics Institute, Department of Engineering Science, University of Oxford（牛津大学机器人研究所、工程科学系、牛津大学）；Dyson Institute of Engineering & Technology（戴森工程与技术研究所）；Bristol Robotics Laboratory, School of Engineering, University of West England（布里斯托尔机器人实验室、工程学院、西英格兰大学）
Developing personal robots that can perform a diverse range of manipulation tasks in unstructured environments necessitates solving several challenges for robotic grasping systems. We take a step towards this broader goal by presenting the first RL-based system, to our knowledge, for a mobile manipulator that can (a) achieve targeted grasping generalizing to unseen target objects, (b) learn complex grasping strategies for cluttered scenes with occluded objects, and (c) perform active vision through its movable wrist camera to better locate objects. The system is informed of the desired target …
AIRoA MoMa Dataset: A Large-Scale Hierarchical Dataset for Mobile Manipulation
The University of Tokyo（东京大学）；AI Robot Association (AIRoA)（人工智能机器人协会）；Toyota Motor Corporation（丰田汽车公司）；Telexistence, Inc.（Telexistence公司）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Waseda University（早稻田大学）
Laboratory for Robotics and Intelligent Control systems（机器人与智能控制系统实验室）；University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）；Unska 3, Zagreb, Croatia（克罗地亚萨格勒布Unska 3号）
In this paper we present our hardware design and control approaches for a mobile manipulation platform used in Challenge 2 of the MBZIRC 2020 competition. In this challenge, a team of UAVs and a single UGV collaborate in an autonomous, wall-building scenario, motivated by construction automation and large-scale robotic 3D printing. The robots must be able, autonomously, to detect, manipulate, and transport bricks in an unstructured, outdoor environment. Our control approach is based on a state machine that dictates which controllers are active at each stage of the Challenge. In the first stage…
Language-Enhanced Mobile Manipulation for Efficient Object Search in Indoor Environments
School of Computation, Information and Technology (CIT), Technical University of Munich（计算、信息与技术学院（CIT），慕尼黑技术大学）；State Key Laboratory for Novel Software Technology and the School of Science and Technology, Nanjing University (Suzhou Campus)（新型软件技术国家重点实验室和南京大学科学技术学院（苏州校区））
University of Surrey（萨里大学）；UK Atomic Energy Authority（英国原子能管理局）
Terrain traversability analysis plays a major role in ensuring safe robotic navigation in unstructured environments. However, real-time constraints frequently limit the accuracy of online tests especially in scenarios where realistic robot-terrain interactions are complex to model. In this context, we propose a deep learning framework trained in an end-to-end fashion from elevation maps and trajectories to estimate the occurrence of failure events. The network is first trained and tested in simulation over synthetic maps generated by the OpenSimplex algorithm. The prediction performance of the…
The Chinese University of Hong Kong（香港中文大学）；The University of British Columbia（不列颠哥伦比亚大学）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
Robots are increasingly operating in indoor environments designed for and shared with people. However, robots working safely and autonomously in uneven and unstructured environments still face great challenges. Many modern indoor environments are designed with wheelchair accessibility in mind. This presents an opportunity for wheeled robots to navigate through sloped areas while avoiding staircases. In this paper, we present an integrated software and hardware system for autonomous mobile robot navigation in uneven and unstructured indoor environments. This modular and reusable software framew…
Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation
The Hong Kong Polytechnic University（香港理工大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Great Bay University（大湾区大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Mobile Manipulation (MM) systems are ideal candidates for taking up the role of a personal assistant in unstructured real-world environments. Among other challenges, MM requires effective coordination of the robot's embodiments for executing tasks that require both mobility and manipulation. Reinforcement Learning (RL) holds the promise of endowing robots with adaptive behaviors, but most methods require prohibitively large amounts of data for learning a useful control policy. In this work, we study the integration of robotic reachability priors in actor-critic RL methods for accelerating the …
Autonomous Legged Mobile Manipulation for Lunar Surface Operations via Constrained Reinforcement Learning
Department of Computer Sciences and Artificial Intelligence, University of Alicante（计算机科学与人工智能系，阿利坎特大学）；Department of Physics, Systems Engineering, and Signal Theory, University of Alicante（物理、系统工程与信号理论系，阿利坎特大学）；Department of Systems and Automatics Engineering, University of Málaga（系统与自动工程系，马拉加大学）
University of Surrey（萨里大学）；UK Atomic Energy Authority（英国原子能管理局）
Driving energy consumption plays a major role in the navigation of mobile robots in challenging environments, especially if they are left to operate unattended under limited on-board power. This paper reports on first results of an energy-aware path planner, which can provide estimates of the driving energy consumption and energy recovery of a robot traversing complex uneven terrains. Energy is estimated over trajectories making use of a self-supervised learning approach, in which the robot autonomously learns how to correlate perceived terrain point clouds to energy consumption and recovery. …
Terrain-Adaptive Mobile 3D Printing with Hierarchical Control
Department of Electrical and Computer Engineering, University of Washington, USA（电气与计算机工程系，华盛顿大学）；Department of Applied Mathematics, University of Washington, USA（应用数学系，华盛顿大学）
Norwegian University of Life Sciences（挪威生命科学大学）；University of Lincoln（林肯大学）
This paper introduces a new method for robot motion planning and navigation in uneven environments through a surfel representation of underlying point clouds. The proposed method addresses the shortcomings of state-of-the-art navigation methods by incorporating both kinematic and physical constraints of a robot with standard motion planning algorithms (e.g., those from the Open Motion Planning Library), thus enabling efficient sampling-based planners for challenging uneven terrain navigation on raw point cloud maps. Unlike techniques based on Digital Elevation Maps (DEMs), our novel surfel-bas…
This paper presents a method for local motion planning in unstructured environments with static and moving obstacles, such as humans. Given a reference path and speed, our optimization-based receding-horizon approach computes a local trajectory that minimizes the tracking error while avoiding obstacles. We build on nonlinear model-predictive contouring control (MPCC) and extend it to incorporate a static map by computing, online, a set of convex regions in free space. We model moving obstacles as ellipsoids and provide a correct bound to approximate the collision region, given by the Minkowsky…
York University（约克大学）；Technische Universität München（慕尼黑工业大学）
We present an active visual search model for finding objects in unknown environments. The proposed algorithm guides the robot towards the sought object using the relevant stimuli provided by the visual sensors. Existing search strategies are either purely reactive or use simplified sensor models that do not exploit all the visual information available. In this paper, we propose a new model that actively extracts visual information via visual attention techniques and, in conjunction with a non-myopic decision-making algorithm, leads the robot to search more relevant areas of the environment. Th…
Artificial Intelligence Research Institute (IIIA-CSIC)（人工智能研究所 (IIIA-CSIC)）；Computer Vision Center（计算机视觉中心）；Australian Centre for Field Robotics（澳大利亚野外机器人中心）；The University of Sydney（悉尼大学）
This paper addresses object perception applied to mobile robotics. Being able to perceive semantically meaningful objects in unstructured environments is a key capability in order to make robots suitable to perform high-level tasks in home environments. However, finding a solution for this task is daunting: it requires the ability to handle the variability in image formation in a moving camera with tight time constraints. The paper brings to attention some of the issues with applying three state of the art object recognition and detection methods in a mobile robotics scenario, and proposes met…
Istituto Italiano di Tecnologia（意大利技术研究院）；Università Cattolica del Sacro Cuore（圣心天主教大学）
Mobile manipulators that combine mobility and manipulability, are increasingly being used for various unstructured application scenarios in the field, e.g. vineyards. Therefore, the coordinated motion of the mobile base and manipulator is an essential feature of the overall performance. In this paper, we explore a whole-body motion controller of a robot which is composed of a 2-DoFs non-holonomic wheeled mobile base with a 7-DoFs manipulator (non-holonomic wheeled mobile manipulator, NWMM) This robotic platform is designed to efficiently undertake complex grapevine pruning tasks. In the contro…
Online Adaptation of Terrain-Aware Dynamics for Planning in Unstructured Environments
Oden Institute for Computational Engineering & Science, University of Texas at Austin（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校）；Department of Computer Science, University of Texas at Austin（计算机科学系，德克萨斯大学奥斯汀分校）；DEVCOM Army Research Laboratory（国防部陆军研究实验室）
German Aerospace Center (DLR)（德国航空航天中心（DLR））；Technical University of Munich（慕尼黑工业大学）
We present the DLR Planetary Stereo, Solid-State LiDAR, Inertial (S3LI) dataset, recorded on Mt. Etna, Sicily, an environment analogous to the Moon and Mars, using a hand-held sensor suite with attributes suitable for implementation on a space-like mobile rover. The environment is characterized by challenging conditions regarding both the visual and structural appearance: severe visual aliasing poses significant limitations to the ability of visual SLAM systems to perform place recognition, while the absence of outstanding structural details, joined with the limited Field-of-View of the utiliz…
Institute of Robotics and Mechatronics（机器人与机电一体化研究所）；German Aerospace Center (DLR)（德国航空航天中心（DLR））；Centre for Autonomous Systems（自主系统中心）；University of Technology Sydney（悉尼科技大学）
Simultaneous Localization and Mapping (SLAM) techniques play a key role towards long-term autonomy of mobile robots due to the ability to correct localization errors and produce consistent maps of an environment over time. Contrarily to urban or man-made environments, where the presence of unique objects and structures offer unique cues for localization, the appearance of unstructured natural environments is often ambiguous and self-similar, hindering the performances of loop closure detection. In this paper, we present an approach to improve the robustness of place recognition in the context …
