Learning to execute long-horizon mobile manipulation tasks is crucial for advancing robotics in household and workplace settings. However, current approaches are typically data-inefficient, underscoring the need for improved models that require realistically sized benchmarks to evaluate their efficiency. To address this, we introduce the LAMBDA (λ) benchmark-Long-horizon Actions for Mobile-manipulation Benchmarking of Directed Activities-which evaluates the data efficiency of models on language-conditioned, long-horizon, multi-room, multi-floor, pick-and-place tasks using a dataset of manageable size, more feasible for collection. Our benchmark includes 571 human-collected demonstrations that provide realism and diversity in simulated and real-world settings. Unlike planner-generated data, these trajectories offer natural variability and replay-verifiability, ensuring robust learning and evaluation. We leverage λ to benchmark current end-to-end learning methods and a modular neuro-symbolic approach that combines foundation models with task and motion planning. We find that learning methods, even when pretrained, yield lower success rates, while a neuro-symbolic method performs significantly better and requires less data.
UC San Diego(加州大学圣地亚哥分校);Meta AI Research(Meta人工智能研究院);UC Berkeley(加州大学伯克利分校)
We study a modular approach to tackle long-horizon mobile manipulation tasks for object rearrangement, which decomposes a full task into a sequence of subtasks. To tackle the entire task, prior work chains multiple stationary manipulation skills with a point-goal navigation skill, which are learned individually on subtasks. Although more effective than monolithic end-to-end RL policies, this framework suffers from compounding errors in skill chaining, e.g., navigating to a bad location where a stationary manipulation skill can not reach its target to manipulate. To this end, we propose that th…
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences(中国科学院机器人与智能制造创新研究院);University of Chinese Academy of Sciences(中国科学院大学);Heriot-Watt University(赫瑞-瓦特大学)
Agile control of mobile manipulator is challenging because of the high complexity coupled by the robotic system and the unstructured working environment. Tracking and grasping a dynamic object with a random trajectory is even harder. In this paper, a multi-task reinforcement learning-based mobile manipulation control framework is proposed to achieve general dynamic object tracking and grasping. Several basic types of dynamic trajectories are chosen as the task training set. To improve the policy generalization in practice, random noise and dynamics randomization are introduced during the train…
We construct a Virtual Kinematic Chain (VKC) that readily consolidates the kinematics of the mobile base, the arm, and the object to be manipulated in mobile manipulations. Accordingly, a mobile manipulation task is represented by altering the state of the constructed VKC, which can be converted to a motion planning problem, formulated, and solved by trajectory optimization. This new VKC perspective of mobile manipulation allows a service robot to (i) produce well-coordinated motions, suitable for complex household environments, and (ii) perform intricate multi-step tasks while interacting wit…
We present a Virtual Kinematic Chain (VKC) perspective, a simple yet effective method, to improve task planning efficacy for mobile manipulation. By consolidating the kinematics of the mobile base, the arm, and the object being manipulated collectively as a whole, this novel VKC perspective naturally defines abstract actions and eliminates unnecessary predicates in describing intermediate poses. As a result, these advantages simplify the design of the planning domain and significantly reduce the search space and branching factors in solving planning problems. In experiments, we implement a tas…
AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);Nanjing University(南京大学);CUHK(香港中文大学);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院)
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences(中国科学院机器人与智能制造创新研究院);University of Chinese Academy of Sciences(中国科学院大学);Westlake University(西湖大学)
Due to their ability to adapt to different terrains, quadruped robots have drawn much attention in the research field of robot learning. Legged mobile manipulation, where a quadruped robot is equipped with a robotic arm, can greatly enhance the performance of the robot in diverse manipulation tasks. Several prior works have investigated legged mobile manipulation from the viewpoint of control theory. However, modeling a unified structure for various robotic arms and quadruped robots is a challenging task. In this paper, we propose a unified framework disturbance predictive control where a rein…
We propose the concept of Action-Related Place (ARPlace) as a powerful and flexible representation of task-related place in the context of mobile manipulation. ARPlace represents robot base locations not as a single position, but rather as a collection of positions, each with an associated probability that the manipulation action will succeed when located there. ARPlaces are generated using a predictive model that is acquired through experience-based learning, and take into account the uncertainty the robot has about its own location and the location of the object to be manipulated. When execu…
PAL Robotics S.L.(PAL机器人有限公司);Bielefeld University(比勒费尔德大学)
Despite many advances in service robotics, successful and secure object manipulation on mobile platforms is still a challenge. In order to come closer to human grasping performance, it is natural to provide robots with the same capability that humans have: the sense of touch. This abstract presents novel, tactile-equipped end-effectors for the service robot TIAGo that are currently being developed. Their primary goal is to improve reliability and success of mobile manipulation, but they also enable further research in related fields such as learning by human demonstration, object exploration a…
MoTo: A Zero-shot Plug-in Interaction-aware Navigation for General Mobile Manipulation
Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);Beijing Key Laboratory of Embodied Intelligence Systems(北京市具身智能系统重点实验室);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Nanyang Technological University(南洋理工大学)
The Omnid human-collaborative mobile manipulators are an experimental platform for testing control architectures for autonomous and human-collaborative multirobot mobile manipulation. An Omnid consists of a mecanum-wheel omnidirectional mobile base and a series-elastic Delta-type parallel manipulator, and it is a specific implementation of a broader class of mobile collaborative robots ("mocobots") suitable for safe human co-manipulation of delicate, flexible, and articulated payloads. Key features of mocobots include passive compliance, for the safety of the human and the payload, and high-fi…
Columbia University(哥伦比亚大学);U.S. Army Research Laboratory(美国陆军研究实验室);Robotics at Google(谷歌机器人部门)
While both navigation and manipulation are challenging topics in isolation, many tasks require the ability to both navigate and manipulate in concert. To this end, we propose a mobile manipulation system that leverages novel navigation and shape completion methods to manipulate an object with a mobile robot. Our system utilizes uncertainty in the initial estimation of a manipulation target to calculate a predicted next-best-view. Without the need of localization, the robot then uses the predicted panoramic view at the next-best-view location to navigate to the desired location, capture a secon…
Queensland University of Technology(昆士兰科技大学);LYRO Robotics(LYRO机器人公司)
We present a generalised architecture for reactive mobile manipulation while a robot's base is in motion toward the next objective in a high-level task. By performing tasks on-the-move, overall cycle time is reduced compared to methods where the base pauses during manipulation. Reactive control of the manipulator enables grasping objects with unpredictable motion while improving robustness against perception errors, environmental disturbances, and inaccurate robot control compared to open-loop, trajectory-based planning approaches. We present an example implementation of the architecture and i…
Analyzing scenes thoroughly is crucial for mobile robots acting in different environments. Semantic segmentation can enhance various subsequent tasks, such as (semantically assisted) person perception, (semantic) free space detection, (semantic) mapping, and (semantic) navigation. In this paper, we propose an efficient and robust RGB-D segmentation approach that can be optimized to a high degree using NVIDIA TensorRT and, thus, is well suited as a common initial processing step in a complex system for scene analysis on mobile robots. We show that RGB-D segmentation is superior to processing RG…
Scene understanding is an active research area. Commercial depth sensors, such as Kinect, have enabled the release of several RGB-D datasets over the past few years which spawned novel methods in 3D scene understanding. More recently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high quality RGB-D data is accessible to millions of people on a device they commonly use. This opens a whole new era in scene understanding for the Computer Vision community as well as app developers. The fundamental research in scene understanding together with the advances in machine learning can…
GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion
CFCS, School of Computer Science, Peking University(北京大学计算机学院CFCS);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Galbot Tongji University(同济大学Galbot)
Stereo Multistage Spatial Attention for Real-Time Mobile Manipulation Under Visual Scale Variation and Disturbances
Department of Intermedia Art and Science, Waseda University(媒体艺术与科学系,早稻田大学);SB Intuitions Corp.(SB Intuitions公司);Research and Development Group, Hitachi, Ltd.(日立株式会社研发部);National Institute of Advanced Industrial Science and Technology (AIST)(国家先进工业科学与技术研究院)
The FetchIt! Mobile Manipulation Challenge, held at the IEEE International Conference on Robots and Automation (ICRA) in May 2019, offered an environment with complex and integrated task sets, irregular objects, confined space, and machining, introducing new challenges in the mobile manipulation domain. Here we describe our efforts to address these challenges by demonstrating the assembly of a kit of mechanical parts in a caddy. In addition to implementation details, we examine the issues in this task set extensively, and we discuss our software architecture in the hope of providing a base for…
Robots are being increasingly used in the fight against highly-infectious diseases such as Ebola, MERS, and SARS-COV-2. Many of the robots that are being used employ ultraviolet lights mounted on a mobile base to inactivate the pathogens. However, these lights are often mounted in a fixed configuration and do not provide adequate decontamination of horizontal surfaces, which can be a major source of cross-contamination. In the paper, we describe the design, implementation, and testing of an Ultraviolet Germicidal Irradiation (UVGI) system implemented on a mobile manipulation robot. A human sup…
This paper presents a framework that allows online dynamic-stability-constrained optimal trajectory planning of a mobile manipulator robot working on rough terrain. First, the kinematics model of a mobile manipulator robot, and the Zero Moment Point (ZMP) stability measure are presented as theoretical background. Then, a sampling-based quasi-static planning algorithm modified for stability guarantee and traction optimization in continuous dynamic motion is presented along with a mathematical proof. The robot's quasi-static path is then used as an initial guess to warm-start a nonlinear optimal…
InCoM: Intent-Driven Perception and Structured Coordination for Mobile Manipulation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);AnyverseDynamics
DynaMOMA: Instantaneous Prediction of Grasp Poses for Mobile Manipulation of Dynamic Objects
National University of Defense Technology(国防科技大学);CFCS, Peking University(北京大学前沿计算研究中心);Defense Innovation Institute, Academy of Military Sciences(军事科学院国防科技创新研究院);Wuhan University(武汉大学);Institute of AI for Industries, Chinese Academy of Sciences(中国科学院产业人工智能研究院)
Cracking Mobile Manipulation: From Failure Diagnosis to Agentic Robot System
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Alibaba DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tsinghua University(清华大学);University of Technology Sydney(悉尼科技大学);University of Liverpool(利物浦大学)
Language-Enhanced Mobile Manipulation for Efficient Object Search in Indoor Environments
School of Computation, Information and Technology (CIT), Technical University of Munich(计算、信息与技术学院(CIT),慕尼黑技术大学);State Key Laboratory for Novel Software Technology and the School of Science and Technology, Nanjing University (Suzhou Campus)(新型软件技术国家重点实验室和南京大学科学技术学院(苏州校区))
Morphologically Equivariant Flow Matching for Bimanual Mobile Manipulation
PEARL Lab, Dept. of Computer Science, TU Darmstadt, Germany(德累斯顿技术大学计算机科学系PEARL实验室);Istituto Italiano di Tecnologia(意大利技术研究院);Robotics Institute Germany(德国机器人研究所)
A particular type of assistive robots designed for physical interaction with objects could play an important role assisting with mobility and fall prevention in healthcare facilities. Autonomous mobile manipulation presents a hurdle prior to safely using robots in real life applications. In this article, we introduce a mobile manipulation framework based on model predictive control using learned dynamics models of objects. We focus on the specific problem of manipulating legged objects such as those commonly found in healthcare environments and personal dwellings (e.g. walkers, tables, chairs)…
Mobile manipulation tasks remain one of the critical challenges for the widespread adoption of autonomous robots in both service and industrial scenarios. While planning approaches are good at generating feasible whole-body robot trajectories, they struggle with dynamic environments as well as the incorporation of constraints given by the task and the environment. On the other hand, dynamic motion models in the action space struggle with generating kinematically feasible trajectories for mobile manipulation actions. We propose a deep reinforcement learning approach to learn feasible dynamic mo…
Learning from demonstrations is a promising paradigm for transferring knowledge to robots. However, learning mobile manipulation tasks directly from a human teacher is a complex problem as it requires learning models of both the overall task goal and of the underlying actions. Additionally, learning from a small number of demonstrations often introduces ambiguity with respect to the intention of the teacher, making it challenging to commit to one model for generalizing the task to new settings. In this paper, we present an approach to learning flexible mobile manipulation action models and tas…
Bikebot manipulation has advantages of the single-track robot mobility and manipulation dexterity. We present a coordinated pose control of mobile manipulation with the stationary bikebot. The challenges of the bikebot manipulation include the limited steering balance capability of the unstable bikebot and kinematic redundancy of the manipulator. We first present the steering balance model to analyze and explore the maximum steering capability to balance the stationary platform. A balancing equilibrium manifold is then proposed to describe the necessary condition to fulfill the simultaneous pl…
This paper addresses the problem of mobile robot manipulation using object detection. Our approach uses detection and control as complimentary functions that learn from real-world interactions. We develop an end-to-end manipulation method based solely on detection and introduce Task-focused Few-shot Object Detection (TFOD) to learn new objects and settings. Our robot collects its own training data and automatically determines when to retrain detection to improve performance across various subtasks (e.g., grasping). Notably, detection training is low-cost, and our robot learns to manipulate new…