将机器人部署在家庭这类开放式非结构化环境中一直是一个长期存在的研究难题。然而，机器人通常仅在封闭的实验室环境中开展研究，且此前的移动操作研究仅限于抓取-移动-放置任务，这可以说只是该领域研究的冰山一角。\n本文提出了Open-World Mobile Manipulation System（开放世界移动操作系统），这是一种全栈式方案，用于应对开放式非结构化环境中真实铰接式物体的操作任务，例如现实世界中的门、橱柜、抽屉和冰箱。该机器人采用自适应学习框架，首先通过行为克隆从少量数据中学习，随后针对训练分布之外的新物体进行在线实践学习。我们还开发了一款低成本的移动操作硬件平台，能够在非结构化环境中实现安全自主的在线自适应，成本约为20000美元。\n在实验中，我们使用了卡内基梅隆大学校园内4栋建筑中的20个铰接式物体。针对每个物体进行不到一小时的在线学习后，该系统能够将成功率从BC预训练的50%提升至采用在线自适应后的95%。视频结果可查看https://open-world-mobilemanip.github.io/
英文摘要：
Deploying robots in open-ended unstructured environments such as homes has been a long-standing research problem. However, robots are often studied only in closed-off lab settings, and prior mobile manipulation work is restricted to pick-move-place, which is arguably just the tip of the iceberg in this area. In this paper, we introduce Open-World Mobile Manipulation System, a full-stack approach to tackle realistic articulated object operation, e.g. real-world doors, cabinets, drawers, and refrigerators in open-ended unstructured environments. The robot utilizes an adaptive learning framework to initially learns from a small set of data through behavior cloning, followed by learning from online practice on novel objects that fall outside the training distribution. We also develop a low-cost mobile manipulation hardware platform capable of safe and autonomous online adaptation in unstructured environments with a cost of around 20,000 USD. In our experiments we utilize 20 articulate objects across 4 buildings in the CMU campus. With less than an hour of online learning for each object, the system is able to increase success rate from 50% of BC pre-training to 95% using online adaptation. Video results at https://open-world-mobilemanip.github.io/
Manipulating articulated objects requires multiple robot arms in general. It is challenging to enable multiple robot arms to collaboratively complete manipulation tasks on articulated objects. In this paper, we present $\textbf{V-MAO}$, a framework for learning multi-arm manipulation of articulated objects. Our framework includes a variational generative model that learns contact point distribution over object rigid parts for each robot arm. The training signal is obtained from interaction with the simulation environment which is enabled by planning and a novel formulation of object-centric co…
Part-Guided 3D RL for Sim2Real Articulated Object Manipulation
Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心（BNRist））；Shanghai AI Laboratory（上海人工智能实验室）；University of California San Diego（加利福尼亚大学圣迭戈分校）；Tsinghua University（清华大学）
Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding
Beijing University of Posts and Telecommunications（北京邮电大学）；School of Computer Science, Peking University（北京大学计算机学院）；School of EECS, Peking University（北京大学电子工程学院）
Online Estimation and Manipulation of Articulated Objects
RIPL-Lab, University of Waterloo, Canada（滑铁卢大学）；Robot Learning Lab, University of Freiburg, Germany（弗赖堡大学）；School of Informatics, University of Edinburgh, UK（爱丁堡大学）
University of Genoa（热那亚大学）；University of Huddersfield（哈德斯菲尔德大学）
Articulated and flexible objects constitute a challenge for robot manipulation tasks but are present in different real-world settings, including home and industrial environments. Current approaches to the manipulation of articulated and flexible objects employ ad hoc strategies to sequence and perform actions on them depending on a number of physical or geometrical characteristics related to those objects, as well as on an a priori classification of target object configurations. In this paper, we propose an action planning and execution framework, which (i) considers abstract representations o…
We introduce the Universal Manipulation Policy Network (UMPNet) -- a single image-based policy network that infers closed-loop action sequences for manipulating arbitrary articulated objects. To infer a wide range of action trajectories, the policy supports 6DoF action representation and varying trajectory length. To handle a diverse set of objects, the policy learns from objects with different articulation structures and generalizes to unseen objects or categories. The policy is trained with self-guided exploration without any human demonstrations, scripted policy, or pre-defined goal conditi…
TONAV: Task-Oriented Navigation and Action-Velocity Chunk Learning for Articulated Object Quadrupedal Mobile Manipulation
School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；College of Semiconductors (College of Integrated Circuits), Hunan University（湖南大学半导体学院（集成电路学院））
FoLD: Force-Informed Learning for Dexterous Articulated Object Manipulation
University of Science and Technology of China（中国科学技术大学）；Institute of AI for Industries, Chinese Academy of Sciences（中国科学院人工智能产业研究院）；Shenzhen University（深圳大学）
University of Genoa（热那亚大学）；Teseo srl（Teseo公司）；University of Huddersfield（哈德斯菲尔德大学）
This paper addresses two intertwined needs for collaborative robots operating in shop-floor environments. The first is the ability to perform complex manipulation operations, such as those on articulated or even flexible objects, in a way robust to a high degree of variability in the actions possibly carried out by human operators during collaborative tasks. The second is encoding in such operations a basic knowledge about physical laws (e.g., gravity), and their effects on the models used by the robot to plan its actions, to generate more robust plans. We adopt the manipulation in three-dimen…
DexSim2Real$^{2}$: Building Explicit World Model for Precise Articulated Object Dexterous Manipulation
Department of Mechanical Engineering, Tsinghua University（清华大学机械工程系）；Institute for Robotics and Intelligent Machines (IRIM), Georgia Institute of Technology（机器人与智能机器研究所（IRIM），佐治亚理工学院）；JD Explore Academy, Beijing, China（京东探索研究院，北京，中国）
Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Beijing Institute for General Artificial Intelligence（北京人工智能研究院）；Department of Automation, Tsinghua University（清华大学自动化系）；College of Engineering, Peking University（北京大学工程学院）；PKU-Wuhan Institute for Artificial Intelligence（北大-武汉人工智能研究院）
Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions
CFCS, School of CS, PKU（计算机学院，北京大学）；School of EECS, PKU（电子工程系，北京大学）；University of Oxford（牛津大学）；National Key Laboratory for Multimedia Information Processing, School of CS, PKU（多媒体信息处理国家重点实验室，计算机学院，北京大学）
Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand
State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University（机械系统与振动国家重点实验室，机械工程学院，上海交通大学）；Shanghai Key Laboratory of Intelligent Robotics, Meta Robotics Institute, Shanghai Jiao Tong University（智能机器人上海重点实验室，元机器人研究院，上海交通大学）
Massachusetts Institute of Technology（麻省理工学院）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Representing and reasoning about uncertainty is crucial for autonomous agents acting in partially observable environments with noisy sensors. Partially observable Markov decision processes (POMDPs) serve as a general framework for representing problems in which uncertainty is an important factor. Online sample-based POMDP methods have emerged as efficient approaches to solving large POMDPs and have been shown to extend to continuous domains. However, these solutions struggle to find long-horizon plans in problems with significant uncertainty. Exploration heuristics can help guide planning, but…
University of Michigan（密歇根大学）；Morehouse College（莫尔豪斯学院）
Articulated objects pose a unique challenge for robotic perception and manipulation. Their increased number of degrees-of-freedom makes tasks such as localization computationally difficult, while also making the process of real-world dataset collection unscalable. With the aim of addressing these scalability issues, we propose Neural Articulated Radiance Fields (NARF22), a pipeline which uses a fully-differentiable, configuration-parameterized Neural Radiance Field (NeRF) as a means of providing high quality renderings of articulated objects. NARF22 requires no explicit knowledge of the object…
Robots operating in domestic environments generally need to interact with articulated objects, such as doors, cabinets, dishwashers or fridges. In this work, we present a novel, probabilistic framework for modeling articulated objects as kinematic graphs. Vertices in this graph correspond to object parts, while edges between them model their kinematic relationship. In particular, we present a set of parametric and non-parametric edge models and how they can robustly be estimated from noisy pose observations. We furthermore describe how to estimate the kinematic structure and how to use the lea…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Southern California（南加州大学）
We propose a probabilistic filtering method which fuses joint measurements with depth images to yield a precise, real-time estimate of the end-effector pose in the camera frame. This avoids the need for frame transformations when using it in combination with visual object tracking methods. Precision is achieved by modeling and correcting biases in the joint measurements as well as inaccuracies in the robot model, such as poor extrinsic camera calibration. We make our method computationally efficient through a principled combination of Kalman filtering of the joint measurements and asynchronous…
In this work, we focus on a novel task of category-level functional hand-object manipulation synthesis covering both rigid and articulated object categories. Given an object geometry, an initial human hand pose as well as a sparse control sequence of object poses, our goal is to generate a physically reasonable hand-object manipulation sequence that performs like human beings. To address such a challenge, we first design CAnonicalized Manipulation Spaces (CAMS), a two-level space hierarchy that canonicalizes the hand poses in an object-centric and contact-centric view. Benefiting from the repr…
University of Oxford（牛津大学）；Hong Kong Polytechnic University（香港理工大学）
Recent works in robotic manipulation through reinforcement learning (RL) or imitation learning (IL) have shown potential for tackling a range of tasks e.g., opening a drawer or a cupboard. However, these techniques generalize poorly to unseen objects. We conjecture that this is due to the high-dimensional action space for joint control. In this paper, we take an alternative approach and separate the task of learning 'what to do' from 'how to do it' i.e., whole-body control. We pose the RL problem as one of determining the skill dynamics for a disembodied virtual manipulator interacting with ar…
This paper presents a framework that allows online dynamic-stability-constrained optimal trajectory planning of a mobile manipulator robot working on rough terrain. First, the kinematics model of a mobile manipulator robot, and the Zero Moment Point (ZMP) stability measure are presented as theoretical background. Then, a sampling-based quasi-static planning algorithm modified for stability guarantee and traction optimization in continuous dynamic motion is presented along with a mathematical proof. The robot's quasi-static path is then used as an initial guess to warm-start a nonlinear optimal…
UC San Diego（加州大学圣地亚哥分校）；Meta AI Research（Meta人工智能研究院）；UC Berkeley（加州大学伯克利分校）
We study a modular approach to tackle long-horizon mobile manipulation tasks for object rearrangement, which decomposes a full task into a sequence of subtasks. To tackle the entire task, prior work chains multiple stationary manipulation skills with a point-goal navigation skill, which are learned individually on subtasks. Although more effective than monolithic end-to-end RL policies, this framework suffers from compounding errors in skill chaining, e.g., navigating to a bad location where a stationary manipulation skill can not reach its target to manipulate. To this end, we propose that th…
Bikebot manipulation has advantages of the single-track robot mobility and manipulation dexterity. We present a coordinated pose control of mobile manipulation with the stationary bikebot. The challenges of the bikebot manipulation include the limited steering balance capability of the unstable bikebot and kinematic redundancy of the manipulator. We first present the steering balance model to analyze and explore the maximum steering capability to balance the stationary platform. A balancing equilibrium manifold is then proposed to describe the necessary condition to fulfill the simultaneous pl…
We construct a Virtual Kinematic Chain (VKC) that readily consolidates the kinematics of the mobile base, the arm, and the object to be manipulated in mobile manipulations. Accordingly, a mobile manipulation task is represented by altering the state of the constructed VKC, which can be converted to a motion planning problem, formulated, and solved by trajectory optimization. This new VKC perspective of mobile manipulation allows a service robot to (i) produce well-coordinated motions, suitable for complex household environments, and (ii) perform intricate multi-step tasks while interacting wit…
Model-Free Large-Scale Cloth Spreading With Mobile Manipulation: Initial Feasibility Study
The Chinese University of Hong Kong（香港中文大学）；Multi-Scale Medical Robotics Center（多尺度医疗机器人中心）；Westlake University（西湖大学）；The Hong Kong University of Science and Technology（香港科技大学）
University of Freiburg（弗莱堡大学）；University of Bremen（不来梅大学）
Service robots in the future need to execute abstract instructions such as "fetch the milk from the fridge". To translate such instructions into actionable plans, robots require in-depth background knowledge. With regards to interactions with doors and drawers, robots require articulation models that they can use for state estimation and motion planning. Existing frameworks model articulated connections as abstract concepts such as prismatic, or revolute, but do not provide a parameterized model of these connections for computation. In this paper, we introduce a novel framework that uses symbo…
GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion
CFCS, School of Computer Science, Peking University（北京大学计算机学院CFCS）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Galbot Tongji University（同济大学Galbot）
We present a Virtual Kinematic Chain (VKC) perspective, a simple yet effective method, to improve task planning efficacy for mobile manipulation. By consolidating the kinematics of the mobile base, the arm, and the object being manipulated collectively as a whole, this novel VKC perspective naturally defines abstract actions and eliminates unnecessary predicates in describing intermediate poses. As a result, these advantages simplify the design of the planning domain and significantly reduce the search space and branching factors in solving planning problems. In experiments, we implement a tas…
