Recent advances in robotics and autonomous systems have broadened the use of robots in laboratory settings, including automated synthesis, scalable reaction workflows, and collaborative tasks in self-driving laboratories (SDLs). This paper presents a comprehensive development of a mobile manipulator designed to assist human operators in such autonomous lab environments. Kinematic modeling of the manipulator is carried out based on the Denavit Hartenberg (DH) convention and inverse kinematics solution is determined to enable precise and adaptive manipulation capabilities. A key focus of this research is enhancing the manipulator ability to reliably grasp textured objects as a critical component of autonomous handling tasks. Advanced vision-based algorithms are implemented to perform real-time object detection and pose estimation, guiding the manipulator in dynamic grasping and following tasks. In this work, we integrate a vision method that combines feature-based detection with homography-driven pose estimation, leveraging depth information to represent an object pose as a $2$D planar projection within $3$D space. This adaptive capability enables the system to accommodate variations in object orientation and supports robust autonomous manipulation across diverse environments. By enabling autonomous experimentation and human-robot collaboration, this work contributes to the scalability and reproducibility of next-generation chemical laboratories
A Bimanual Gesture Interface for ROS-Based Mobile Manipulators Using TinyML and Sensor Fusion
†‡Department of Mechatronics Engineering, Rajshahi University of Engineering \& Technology, Kazla, Rajshahi-6204, Bangladesh §Department of Electrical \& Systems Engineering, School of Engineering \& Applied Science, University of Pennsylvania, Philadelphia, PA 19104, United States Email: , †, ‡, §
University of Toronto Institute for Aerospace Studies (UTIAS)（多伦多大学航空航天研究所（UTIAS））；Vector Institute for artificial Intelligence（人工智能向量研究所）；University of Toronto（多伦多大学）
Mobile manipulators consist of a mobile platform equipped with one or more robot arms and are of interest for a wide array of challenging tasks because of their extended workspace and dexterity. Typically, mobile manipulators are deployed in slow-motion collaborative robot scenarios. In this paper, we consider scenarios where accurate high-speed motions are required. We introduce a framework for this regime of tasks including two main components: (i) a bi-level motion optimization algorithm for real-time trajectory generation, which relies on Sequential Quadratic Programming (SQP) and Quadrati…
Whole-body Motion Control of an Omnidirectional Wheel-Legged Mobile Manipulator via Contact-Aware Dynamic Optimization
State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology（智能制造装备与技术国家重点实验室，机械科学与工程学院，华中科技大学）
This paper presents a system integration approach for a 6-DoF (Degree of Freedom) collaborative robot to operate a pipette for liquid dispensing. Its technical development is threefold. First, we designed an end-effector for holding and triggering manual pipettes. Second, we took advantage of a collaborative robot to recognize labware poses and planned robotic motion based on the recognized poses. Third, we developed vision-based classifiers to predict and correct positioning errors and thus precisely attached pipettes to disposable tips. Through experiments and analysis, we confirmed that the…
Multi-Sensor Fusion-Based Mobile Manipulator Remote Control for Intelligent Smart Home Assistance
Dyson School of Design Engineering, Imperial College London（帝国理工学院伦敦设计工程学院）；School of Robotics, Xi’an Jiaotong-Liverpool University (XJTLU)（西安交通大学利物浦大学机器人学院）；Department of Engineering, University of Cambridge（剑桥大学工程学院）；School of Mechanical Engineering, Northwestern Polytechnical University（西北工业大学机械工程学院）；Department of Computer Science, University of Aberdeen（阿伯丁大学计算机科学学院）
University College London（伦敦大学学院）；University of Ioannina（约阿尼纳大学）
Many robotic path planning problems are continuous, stochastic, and high-dimensional. The ability of a mobile manipulator to coordinate its base and manipulator in order to control its whole-body online is particularly challenging when self and environment collision avoidance is required. Reinforcement Learning techniques have the potential to solve such problems through their ability to generalise over environments. We study joint penalties and joint limits of a state-of-the-art mobile manipulator whole-body controller that uses LIDAR sensing for obstacle collision avoidance. We propose direc…
Socially-aware Object Transportation by a Mobile Manipulator in Static Planar Environments with Obstacles
Computer Vision and Robotics Laboratory (VeRLab), Department of Computer Science, Universidade Federal de Minas Gerais（计算机视觉与机器人实验室（VeRLab）、计算机科学系、巴西矿务大学）
Existing approaches to direct robot-to-human handovers are typically implemented on fixed-base robot arms, or on mobile manipulators that come to a full stop before performing the handover. We propose "on-the-go" handovers which permit a moving mobile manipulator to hand over an object to a human without stopping. The on-the-go handover motion is generated with a reactive controller that allows simultaneous control of the base and the arm. In a user study, human receivers subjectively assessed on-the-go handovers to be more efficient, predictable, natural, better timed and safer than handovers…
This work addresses the problem of kinematic trajectory planning for mobile manipulators with non-holonomic constraints, and holonomic operational-space tracking constraints. We obtain whole-body trajectories and time-varying kinematic feedback controllers by solving a Constrained Sequential Linear Quadratic Optimal Control problem. The employed algorithm features high efficiency through a continuous-time formulation that benefits from adaptive step-size integrators and through linear complexity in the number of integration steps. In a first application example, we solve kinematic trajectory p…
Mobile manipulator throwing is a promising method to increase the flexibility and efficiency of dynamic manipulation in factories. Its major challenge is to efficiently plan a feasible throw under a wide set of task specifications. We show that the mobile manipulator throwing problem can be simplified to a planar problem, hence greatly reducing the computational costs. Using machine learning approaches, we build a model of the object's inverted flying dynamics and the robot's kinematic feasibility, which enables throwing motion generation within 1 ms for given query of target position. Thanks …
Örebro University（厄勒布鲁大学）；Center for Applied Autonomous Sensor Systems (AASS)（应用自主传感器系统中心）
Integrating the heterogeneous controllers of a complex mechanical system, such as a mobile manipulator, within the same structure and in a modular way is still challenging. In this work we extend our framework based on Behavior Trees for the control of a redundant mechanical system to the problem of commanding more complex systems that involve multiple low-level controllers. This allows the integrated systems to achieve non-trivial goals that require coordination among the sub-systems.
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；University of Zagreb（萨格勒布大学）
We present a novel approach for mobile manipulator self-calibration using contact information. Our method, based on point cloud registration, is applied to estimate the extrinsic transform between a fixed vision sensor mounted on a mobile base and an end effector. Beyond sensor calibration, we demonstrate that the method can be extended to include manipulator kinematic model parameters, which involves a non-rigid registration process. Our procedure uses on-board sensing exclusively and does not rely on any external measurement devices, fiducial markers, or calibration rigs. Further, it is full…
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology (AIST)（国立先进工业科学技术研究所（AIST））
In this paper, we present a planner that plans a sequence of base positions for a mobile manipulator to efficiently and robustly collect objects stored in distinct trays. We achieve high efficiency by exploring the common areas where a mobile manipulator can grasp objects stored in multiple trays simultaneously and move the mobile manipulator to the common areas to reduce the time needed for moving the mobile base. We ensure robustness by optimizing the base position with the best clearance to positioning uncertainty so that a mobile manipulator can complete the task even if there is a certain…
HP-NTU Digital Manufacturing Corporate Lab（惠普-南洋理工大学数字制造企业实验室）；Nanyang Technological University（南洋理工大学）
We introduce a new approach to tackle the mobile manipulator task sequencing problem. We leverage computational geometry, graph theory and combinatorial optimization to yield a principled method to segment the task-space targets into clusters, analytically determine reachable base pose for each cluster, and find task sequences that minimize the number of base movements and robot execution time. By clustering targets first and by doing so from first principles, our solution is more general and computationally efficient when compared to existing methods.
University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
Human-robot collaborative tasks foresee interactions between humans and robots with various degrees of complexity. Specifically, for tasks which involve physical contact among the agents, challenges arise in the modelling and control of such interaction. In this paper we propose a control architecture capable of ensuring a flexible and robustly stable physical human-robot interaction, focusing on a collaborative transportation task. The architecture is deployed onto a mobile manipulator, modelled as a whole-body structure, which aids the operator during the transportation of an unwieldy load. …
Adaptive falling and recovery skills greatly extend the applicability of robot deployments. In the case of legged mobile manipulators, the robot arm could adaptively stop the fall and assist the recovery. Prior works on falling and recovery strategies for legged mobile manipulators usually rely on assumptions such as inelastic collisions and falling in defined directions to enable real-time computation. This paper presents a learning-based approach to reducing fall damage and recovery. An asymmetric actor-critic training structure is used to train a time-invariant policy with time-varying rewa…
Task-Space Clustering for Mobile Manipulator Task Sequencing
Singapore Centre for 3D Printing (SC3DP)（新加坡3D打印中心（SC3DP））；Nanyang Technological University (NTU)（南洋理工大学（NTU））；HP-NTU Digital Manufacturing Corporate Lab（惠普-南洋理工大学数字制造联合实验室）；Eureka Robotics（优傲机器人（Eureka Robotics））
IIIT Hyderabad（国际信息技术研究所海得拉巴分校）；University of Genoa（热那亚大学）；University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；IIT Jodhpur（焦特布尔印度理工学院）
In this paper, a kinematic motion planning algorithm for cooperative spatial payload manipulation is presented. A hierarchical approach is introduced to compute real-time collision-free motion plans for a formation of mobile manipulator robots. Initially, collision-free configurations of a deformable 2-D virtual bounding box are identified, over a planning horizon, to define a convex workspace for the entire system. Then, 3-D payload configurations whose projections lie within the defined convex workspace are computed. Finally, a convex decentralized model-predictive controller is formulated t…
Autonomous mobile manipulation offers a dual advantage of mobility provided by a mobile platform and dexterity afforded by the manipulator. In this paper, we present a whole-body optimal control framework to jointly solve the problems of manipulation, balancing and interaction as one optimization problem for an inherently unstable robot. The optimization is performed using a Model Predictive Control (MPC) approach; the optimal control problem is transcribed at the end-effector space, treating the position and orientation tasks in the MPC planner, and skillfully planning for end-effector contac…
HEATS: A Hierarchical Framework for Efficient Autonomous Target Search with Mobile Manipulators
School of Mechanical Engineering and Automation, Harbin Institute of Technology Shenzhen（哈尔滨工业大学（深圳）机械工程与自动化学院）；University of Science and Technology of China（中国科学技术大学）
This paper investigates a fully distributed cooperation scheme for networked mobile manipulators. To achieve cooperative task allocation in a distributed way, an adaptation-based estimation law is established for each robotic agent to estimate the desired local trajectory. In addition, wrench synthesis is analyzed in detail to lay a solid foundation for tight cooperation tasks. Together with the estimated task, a set of distributed adaptive controllers is proposed to achieve motion synchronization of the mobile manipulator ensemble over a directed graph with a spanning tree irrespective of the…
Robotic systems performing end-user oriented autonomous exploration can be deployed in different scenarios which not only require mapping but also simultaneous inspection of regions of interest for the end-user. In this work, we propose a novel Next-Best-View (NBV) planner which can perform full exploration and user-oriented exploration with inspection of the regions of interest using a mobile manipulator robot. We address the exploration-inspection problem as an instance of Multi-Objective Optimization (MOO) and propose a weighted-sum-based information gain function for computing NBVs for the…
Deep reinforcement learning produces robust locomotion policies for legged robots over challenging terrains. To date, few studies have leveraged model-based methods to combine these locomotion skills with the precise control of manipulators. Here, we incorporate external dynamics plans into learning-based locomotion policies for mobile manipulation. We train the base policy by applying a random wrench sequence on the robot base in simulation and adding the noisified wrench sequence prediction to the policy observations. The policy then learns to counteract the partially-known future disturbanc…
Federal University of Minas Gerais（米纳斯吉拉斯联邦大学）；The University of Manchester（曼彻斯特大学）
This paper presents two approaches to obtain the dynamical equations of mobile manipulators using dual quaternion algebra. The first one is based on a general recursive Newton-Euler formulation and uses twists and wrenches, which are propagated through high-level algebraic operations and works for any type of joints and arbitrary parameterizations. The second approach is based on Gauss's Principle of Least Constraint (GPLC) and includes arbitrary equality constraints. In addition to showing the connections of GPLC with Gibbs-Appell and Kane's equations, we use it to model a nonholonomic mobile…
Czech Technical University in Prague（布拉格捷克理工大学）；New York University（纽约大学）；University of Pennsylvania（宾夕法尼亚大学）
Mobile manipulators have the potential to revolutionize modern agriculture, logistics and manufacturing. In this work, we present the design of a ground-based mobile manipulator for automated structure assembly. The proposed system is capable of autonomous localization, grasping, transportation and deployment of construction material in a semi-structured environment. Special effort was put into making the system invariant to lighting changes, and not reliant on external positioning systems. Therefore, the presented system is self-contained and capable of operating in outdoor and indoor conditi…
TopAY: Efficient Trajectory Planning for Differential Drive Mobile Manipulators via Topological Paths Search and Arc Length-Yaw Parameterization
State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou 310027, China（浙江大学工业控制技术状态重点实验室）；Huzhou Institute of Zhejiang University, Huzhou 313000, China（浙江大学湖州研究院）
Italian Institute of Technology（意大利技术研究院）；University of Genoa（热那亚大学）
This paper presents a simplified model-based trajectory optimization (TO) formulation for motion planning on quadruped mobile manipulators that carry heavy payload of known mass. The proposed payload-aware formulation simultaneously plans locomotion, payload manipulation and considers both robot and payload model dynamics while remaining computationally efficient. At the presence of heavy payload, the approach exhibits reduced leg outstretching (thus increased manipulability) in kinematically demanding motions due to the contribution of payload manipulation in the optimization. The framework's…
National Institute of Advanced Industrial Science and Technology (AIST)（国立先进工业科学技术研究所（AIST））；Osaka University（大阪大学）
In this paper, we address efficiently and robustly collecting objects stored in different trays using a mobile manipulator. A resolution complete method, based on precomputed reachability database, is proposed to explore collision-free inverse kinematics (IK) solutions and then a resolution complete set of feasible base positions can be determined. This method approximates a set of representative IK solutions that are especially helpful when solving IK and checking collision are treated separately. For real world applications, we take into account the base positioning uncertainty and plan a se…
Ecole Centrale de Lille（里尔中央理工学院）；Beihang University（北京航空航天大学）
A high redundant non-holonomic humanoid mobile dual-arm manipulator system is presented in this paper where the motion planning to realize "human-like" autonomous navigation and manipulation tasks is studied. Firstly, an improved MaxiMin NSGA-II algorithm, which optimizes five objective functions to solve the problems of singularity, redundancy, and coupling between mobile base and manipulator simultaneously, is proposed to design the optimal pose to manipulate the target object. Then, in order to link the initial pose and that optimal pose, an off-line motion planning algorithm is designed. I…
The COVID-19 pandemic has demonstrated the need for a more effective and efficient disinfection approach to combat infectious diseases. Ultraviolet germicidal irradiation (UVGI) is a proven mean for disinfection and sterilization and has been integrated into handheld devices and autonomous mobile robots. Existing UVGI robots which are commonly equipped with uncovered lamps that emit intense ultraviolet radiation suffer from: inability to be used in human presence, shadowing of objects, and long disinfection time. These robots also have a high operational cost. This paper introduces a cost-effe…
A resolved acceleration control (RAC) and proportional-integral active force control (PIAFC) is proposed as an approach for the robust motion control of a mobile manipulator (MM) comprising a differentially driven wheeled mobile platform with a two-link planar arm mounted on top of the platform. The study emphasizes on the integrated kinematic and dynamic control strategy in which the RAC is used to manipulate the kinematic component while the PIAFC is implemented to compensate the dynamic effects including the bounded known/unknown disturbances and uncertainties. The effectivenss and robustne…
Multimodal Control of Manipulators: Coupling Kinematics and Vision for Self-Driving Laboratory Operations
1 Department of Electronics Systems, Aalborg University, Denmark；3 Extreme Robotics Laboratory, School of Metallurgy \& Materials, University of Birmingham, Birmingham, United Kingdom
Seconda Università degli Studi di Napoli（那不勒斯第二大学）
A reactive obstacle avoidance method for mobile manipulators is presented. The objectives of the developed algorithm are twofold. The first one is to find a trajectory in the configuration space of a mobile manipulator so as to follow a given trajectory in the task space. The second objective consists in locally adjusting the trajectory in the configuration space in order to avoid collisions with potentially moving obstacles and self-collisions in unstructured and dynamic environments. The perception is exclusively based on a set of proximity sensors distributed on the robot mechanical structu…
National Key Laboratory of Digital Control and System Engineering (DCSELab), Ho Chi Minh City University of Technology (HCMUT)（胡志明市理工大学数字控制与系统工程国家重点实验室）；Ho Chi Minh City University of Technology (HCMUT)（胡志明市理工大学）；Vietnam National University Ho Chi Minh City（越南国立大学胡志明市分校）；Ton Duc Thang University（孙德胜大学）
All flames are extinguished as early as possible, or fire services have to deal with major conflagrations. This leads to the fact that the quality of fire extinguishers has become a very sensitive and important issue in firefighting. Inspired by the development of automatic fire fighting systems, this paper proposes key specifications based on the standard of fire extinguishers that is ISO 7165:2009 and ISO 11601:2008, and feasible solutions to design a mobile manipulator for automatically evaluating the quality or, more specifically, power of fire extinguishers. In addition, a part of the mec…
