Online Estimation and Manipulation of Articulated Objects
RIPL-Lab, University of Waterloo, Canada（滑铁卢大学）；Robot Learning Lab, University of Freiburg, Germany（弗赖堡大学）；School of Informatics, University of Edinburgh, UK（爱丁堡大学）
Russell Buchanan, Adrian Röfer, João Moura, Abhinav Valada, Sethu Vijayakumar
备注：
This preprint has not undergone peer review or any post-submission improvements or corrections. The Version of Record of this article is published in Autonomous Robots, and is available online at [Link will be updated when available]
From refrigerators to kitchen drawers, humans interact with articulated objects effortlessly every day while completing household chores. For automating these tasks, service robots must be capable of manipulating arbitrary articulated objects. Recent deep learning methods have been shown to predict valuable priors on the affordance of articulated objects from vision. In contrast, many other works estimate object articulations by observing the articulation motion, but this requires the robot to already be capable of manipulating the object. In this article, we propose a novel approach combining these methods by using a factor graph for online estimation of articulation which fuses learned visual priors and proprioceptive sensing during interaction into an analytical model of articulation based on Screw Theory. With our method, a robotic system makes an initial prediction of articulation from vision before touching the object, and then quickly updates the estimate from kinematic and force sensing during manipulation. We evaluate our method extensively in both simulations and real-world robotic manipulation experiments. We demonstrate several closed-loop estimation and manipulation experiments in which the robot was capable of opening previously unseen drawers. In real hardware experiments, the robot achieved a 75% success rate for autonomous opening of unknown articulated objects.
SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining
Institute of Artificial Intelligence, China Telecom（中国电信人工智能研究院）；Technical University of Munich（慕尼黑工业大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）
University of Genoa（热那亚大学）；University of Huddersfield（哈德斯菲尔德大学）
Articulated and flexible objects constitute a challenge for robot manipulation tasks but are present in different real-world settings, including home and industrial environments. Current approaches to the manipulation of articulated and flexible objects employ ad hoc strategies to sequence and perform actions on them depending on a number of physical or geometrical characteristics related to those objects, as well as on an a priori classification of target object configurations. In this paper, we propose an action planning and execution framework, which (i) considers abstract representations o…
University of Genoa（热那亚大学）；Teseo srl（Teseo公司）；University of Huddersfield（哈德斯菲尔德大学）
This paper addresses two intertwined needs for collaborative robots operating in shop-floor environments. The first is the ability to perform complex manipulation operations, such as those on articulated or even flexible objects, in a way robust to a high degree of variability in the actions possibly carried out by human operators during collaborative tasks. The second is encoding in such operations a basic knowledge about physical laws (e.g., gravity), and their effects on the models used by the robot to plan its actions, to generate more robust plans. We adopt the manipulation in three-dimen…
FoLD: Force-Informed Learning for Dexterous Articulated Object Manipulation
University of Science and Technology of China（中国科学技术大学）；Institute of AI for Industries, Chinese Academy of Sciences（中国科学院人工智能产业研究院）；Shenzhen University（深圳大学）
DexSim2Real$^{2}$: Building Explicit World Model for Precise Articulated Object Dexterous Manipulation
Department of Mechanical Engineering, Tsinghua University（清华大学机械工程系）；Institute for Robotics and Intelligent Machines (IRIM), Georgia Institute of Technology（机器人与智能机器研究所（IRIM），佐治亚理工学院）；JD Explore Academy, Beijing, China（京东探索研究院，北京，中国）
Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Beijing Institute for General Artificial Intelligence（北京人工智能研究院）；Department of Automation, Tsinghua University（清华大学自动化系）；College of Engineering, Peking University（北京大学工程学院）；PKU-Wuhan Institute for Artificial Intelligence（北大-武汉人工智能研究院）
Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand
State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University（机械系统与振动国家重点实验室，机械工程学院，上海交通大学）；Shanghai Key Laboratory of Intelligent Robotics, Meta Robotics Institute, Shanghai Jiao Tong University（智能机器人上海重点实验室，元机器人研究院，上海交通大学）
Massachusetts Institute of Technology（麻省理工学院）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Representing and reasoning about uncertainty is crucial for autonomous agents acting in partially observable environments with noisy sensors. Partially observable Markov decision processes (POMDPs) serve as a general framework for representing problems in which uncertainty is an important factor. Online sample-based POMDP methods have emerged as efficient approaches to solving large POMDPs and have been shown to extend to continuous domains. However, these solutions struggle to find long-horizon plans in problems with significant uncertainty. Exploration heuristics can help guide planning, but…
TONAV: Task-Oriented Navigation and Action-Velocity Chunk Learning for Articulated Object Quadrupedal Mobile Manipulation
School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；College of Semiconductors (College of Integrated Circuits), Hunan University（湖南大学半导体学院（集成电路学院））
ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulation
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人学院）；IIIS, Tsinghua University（清华大学人工智能研究所）；Department of Computer Science, National Yang Ming Chiao Tung University（国立阳明交通大学计算机科学系）
Robots operating in domestic environments generally need to interact with articulated objects, such as doors, cabinets, dishwashers or fridges. In this work, we present a novel, probabilistic framework for modeling articulated objects as kinematic graphs. Vertices in this graph correspond to object parts, while edges between them model their kinematic relationship. In particular, we present a set of parametric and non-parametric edge models and how they can robustly be estimated from noisy pose observations. We furthermore describe how to estimate the kinematic structure and how to use the lea…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Southern California（南加州大学）
We propose a probabilistic filtering method which fuses joint measurements with depth images to yield a precise, real-time estimate of the end-effector pose in the camera frame. This avoids the need for frame transformations when using it in combination with visual object tracking methods. Precision is achieved by modeling and correcting biases in the joint measurements as well as inaccuracies in the robot model, such as poor extrinsic camera calibration. We make our method computationally efficient through a principled combination of Kalman filtering of the joint measurements and asynchronous…
University of Michigan（密歇根大学）；Morehouse College（莫尔豪斯学院）
Articulated objects pose a unique challenge for robotic perception and manipulation. Their increased number of degrees-of-freedom makes tasks such as localization computationally difficult, while also making the process of real-world dataset collection unscalable. With the aim of addressing these scalability issues, we propose Neural Articulated Radiance Fields (NARF22), a pipeline which uses a fully-differentiable, configuration-parameterized Neural Radiance Field (NeRF) as a means of providing high quality renderings of articulated objects. NARF22 requires no explicit knowledge of the object…
University of Freiburg（弗莱堡大学）；University of Bremen（不来梅大学）
Service robots in the future need to execute abstract instructions such as "fetch the milk from the fridge". To translate such instructions into actionable plans, robots require in-depth background knowledge. With regards to interactions with doors and drawers, robots require articulation models that they can use for state estimation and motion planning. Existing frameworks model articulated connections as abstract concepts such as prismatic, or revolute, but do not provide a parameterized model of these connections for computation. In this paper, we introduce a novel framework that uses symbo…
End-to-end learning robotic manipulation with high data efficiency is one of the key challenges in robotics. The latest methods that utilize human demonstration data and unsupervised representation learning has proven to be a promising direction to improve RL learning efficiency. The use of demonstration data also allows "warming-up" the RL policies using offline data with imitation learning or the recently emerged offline reinforcement learning algorithms. However, existing works often treat offline policy learning and online exploration as two separate processes, which are often accompanied …
UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models
Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）；Rutgers University（罗格斯大学）；Hefei University of Technology（合肥工业大学）；Shanghai AI Lab（上海人工智能实验室）
Virtualizing the physical world into virtual models has been a critical technique for robot navigation and planning in the real world. To foster manipulation with articulated objects in everyday life, this work explores building articulation models of indoor scenes through a robot's purposeful interactions in these scenes. Prior work on articulation reasoning primarily focuses on siloed objects of limited categories. To extend to room-scale environments, the robot has to efficiently and effectively explore a large-scale 3D space, locate articulated objects, and infer their articulations. We in…
AdaManip: Adaptive Articulated Object Manipulation Environments and Policy Learning
Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Inst. for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Nat’l Eng. Research Center of Visual Technology, Peking University（北京大学视觉技术国家工程研究中心）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Integration of Robot and Scene Kinematics for Sequential Mobile Manipulation Planning
State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing（国家一般人工智能重点实验室，北京一般人工智能研究院（BIGAI），北京）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Institute for Artificial Intelligence and School of Artificial Intelligence, Peking University（北京大学人工智能研究院和人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）
Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Institute of Artificial Intelligence, China Telecom Corp Ltd（中国电信股份有限公司人工智能研究院）
Manipulating articulated objects requires multiple robot arms in general. It is challenging to enable multiple robot arms to collaboratively complete manipulation tasks on articulated objects. In this paper, we present $\textbf{V-MAO}$, a framework for learning multi-arm manipulation of articulated objects. Our framework includes a variational generative model that learns contact point distribution over object rigid parts for each robot arm. The training signal is obtained from interaction with the simulation environment which is enabled by planning and a novel formulation of object-centric co…
GSAM: A Generalizable and Safe Robotic Framework for Articulated Object Manipulation
College of Computer Science, Chongqing University, Chongqing, China（重庆大学计算机学院）；Lumos Robotics, China（Lumos机器人中国）；Xi'an Jiaotong-Liverpool University, China（西安交通大学利物浦大学）；Fudan University, China（复旦大学）；Department of Information Engineering and Computer Science, University of Trento, Trento, Italy（特伦托大学信息工程与计算机科学系）
GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CFCS, School of Computer Science, Peking University（北京大学计算机科学系）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Galbot
In this project we present a framework for building generalizable manipulation controller policies that map from raw input point clouds and segmentation masks to joint velocities. We took a traditional robotics approach, using point cloud processing, end-effector trajectory calculation, inverse kinematics, closed-loop position controllers, and behavior trees. We demonstrate our framework on four manipulation skills on common household objects that comprise the SAPIEN ManiSkill Manipulation challenge.
University of Calabria（卡拉布里亚大学）；Teseo srl（特塞奥有限公司）；University of Genoa（热那亚大学）；University of Huddersfield（哈德斯菲尔德大学）
The manipulation of articulated objects is of primary importance in Robotics, and can be considered as one of the most complex manipulation tasks. Traditionally, this problem has been tackled by developing ad-hoc approaches, which lack flexibility and portability. In this paper we present a framework based on Answer Set Programming (ASP) for the automated manipulation of articulated objects in a robot control architecture. In particular, ASP is employed for representing the configuration of the articulated object, for checking the consistency of such representation in the knowledge base, and f…
Implicit Articulated Robot Morphology Modeling with Configuration Space Neural Signed Distance Functions
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；University College London（伦敦大学学院）；Rice University（莱斯大学）；Massachusetts Institute of Technology（麻省理工学院）
Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）
The deformable linear objects (DLOs) are common in both industrial and domestic applications, such as wires, cables, ropes. Because of its highly deformable nature, it is difficult for the robot to reproduce human's dexterous skills on DLOs. In this paper, the unknown deformation model is estimated in both the offline and online manners. The offline learning aims to provide a good approximation prior to the manipulation task, while the online learning aims to compensate the errors due to insufficient training (e.g. limited datasets) in the offline phase. The offline module works by constructin…
Georgia Institute of Technology（佐治亚理工学院）；Stanford University（斯坦福大学）
We explore the problem of estimating the mass distribution of an articulated object by an interactive robotic agent. Our method predicts the mass distribution of an object by using the limited sensing and actuating capabilities of a robotic agent that is interacting with the object. We are inspired by the role of exploratory play in human infants. We take the combined approach of supervised and reinforcement learning to train an agent that learns to strategically interact with the object to estimate the object's mass distribution. Our method consists of two neural networks: (i) the policy netw…
Part-Guided 3D RL for Sim2Real Articulated Object Manipulation
Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心（BNRist））；Shanghai AI Laboratory（上海人工智能实验室）；University of California San Diego（加利福尼亚大学圣迭戈分校）；Tsinghua University（清华大学）
