Robot learning is a very promising topic for the future of automation and machine intelligence. Future robots should be able to autonomously acquire skills, learn to represent their environment, and interact with it. While these topics have been explored in simulation, real-world robot learning research seems to be still limited. This is due to the additional challenges encountered in the real-world, such as noisy sensors and actuators, safe exploration, non-stationary dynamics, autonomous environment resetting as well as the cost of running experiments for long periods of time. Unless we develop scalable solutions to these problems, learning complex tasks involving hand-eye coordination and rich contacts will remain an untouched vision that is only feasible in controlled lab environments. We propose dual-arm settings as platforms for robot learning. Such settings enable safe data collection for acquiring manipulation skills as well as training perception modules in a robot-supervised manner. They also ease the processes of resetting the environment. Furthermore, adversarial learning could potentially boost the generalization capability of robot learning methods by maximizing the exploration based on game-theoretic objectives while ensuring safety based on collaborative task spaces. In this paper, we will discuss the potential benefits of this setup as well as the challenges and research directions that can be pursued.
Accenture Labs（埃森哲实验室）；National Research Council Canada（加拿大国家研究委员会）
In this paper, we explore the ability of a robot arm to learn the underlying operation space defined by the positions (x, y, z) that the arm's end-effector can reach, including disturbances, by deploying and thoroughly evaluating a Spiking Neural Network SNN-based adaptive control algorithm. While traditional control algorithms for robotics have limitations in both adapting to new and dynamic environments, we show that the robot arm can learn the operational space and complete tasks faster over time. We also demonstrate that the adaptive robot control algorithm based on SNNs enables a fast res…
Brain-computer interface (BCI) uses brain signals to communicate with external devices without actual control. Particularly, BCI is one of the interfaces for controlling the robotic arm. In this study, we propose a knowledge distillation-based framework to manipulate robotic arm through hybrid paradigm induced EEG signals for practical use. The teacher model is designed to decode input data hierarchically and transfer knowledge to student model. To this end, soft labels and distillation loss functions are applied to the student model training. According to experimental results, student model a…
Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging -- asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges,…
Hanson Robotics Ltd（汉森机器人有限公司）；Indian Institute of Science（印度科学学院）；ARTPARK（ARTPARK（人工智能与机器人技术园区））
Open Arms is a novel open-source platform of realistic human-like robotic hands and arms hardware with 28 Degree-of-Freedom (DoF), designed to extend the capabilities and accessibility of humanoid robotic grasping and manipulation. The Open Arms framework includes an open SDK and development environment, simulation tools, and application development tools to build and operate Open Arms. This paper describes these hands controls, sensing, mechanisms, aesthetic design, and manufacturing and their real-world applications with a teleoperated nursing robot. From 2015 to 2022, the authors have desig…
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)
The University of Hong Kong（香港大学）；AgileX Robotics（极目机器人）；Shanghai AI Laboratory（上海人工智能实验室）；Shenzhen University（深圳大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
University of California, Berkeley（加州大学伯克利分校）；Google（谷歌）
Autonomous fabric manipulation is a longstanding challenge in robotics, but evaluating progress is difficult due to the cost and diversity of robot hardware. Using Reach, a cloud robotics platform that enables low-latency remote execution of control policies on physical robots, we present the first systematic benchmarking of fabric manipulation algorithms on physical hardware. We develop 4 novel learning-based algorithms that model expert actions, keypoints, reward functions, and dynamic motions, and we compare these against 4 learning-free and inverse dynamics algorithms on the task of foldin…
Humans can leverage physical interaction to teach robot arms. As the human kinesthetically guides the robot through demonstrations, the robot learns the desired task. While prior works focus on how the robot learns, it is equally important for the human teacher to understand what their robot is learning. Visual displays can communicate this information; however, we hypothesize that visual feedback alone misses out on the physical connection between the human and robot. In this paper we introduce a novel class of soft haptic displays that wrap around the robot arm, adding signals without affect…
A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot
Department of Mechanical Engineering, University of São Paulo（机械工程系，圣保罗大学）；Instituto Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein（艾利哈德研究所，以色列医院）
Shanghai Jiao Tong University（上海交通大学）；University of Oxford（牛津大学）；Tencent（腾讯）
Collision avoidance is a widely investigated topic in robotic applications. When applying collision avoidance techniques to a mobile robot, how to deal with the spatial structure of the robot still remains a challenge. In this paper, we design a configuration-aware safe control law by solving a Quadratic Programming (QP) with designed Control Barrier Functions (CBFs) constraints, which can safely navigate a mobile robotic arm to a desired region while avoiding collision with environmental obstacles. The advantage of our approach is that it correctly and in an elegant way incorporates the spati…
Beijing Academy of Agriculture and Forestry Sciences（北京市农林科学院）；Jiangsu University（江苏大学）
The emergence of harvesting robotics offers a promising solution to the issue of limited agricultural labor resources and the increasing demand for fruits. Despite notable advancements in the field of harvesting robotics, the utilization of such technology in orchards is still limited. The key challenge is to improve operational efficiency. Taking into account inner-arm conflicts, couplings of DoFs, and dynamic tasks, we propose a task planning strategy for a harvesting robot with four arms in this paper. The proposed method employs a Markov game framework to formulate the four-arm robotic har…
This paper uses Model Predictive Control (MPC) to optimise the input torques of a Three-Degrees-of-Freedom (DOF) robotic arm, enabling it to operate to the target position and grasp the object accurately. A monocular camera is firstly used to recognise the colour and depth of the object. Then, the inverse kinematics calculation and the spatial coordinates of the object through coordinate transformation are combined to get the required rotating angle of each servo. Finally, the dynamic model of the robotic arm structure is derived and the model predictive control is applied to simulate the opti…
Osaka University（大阪大学）；National Inst. of AIST（国立先进工业科学技术研究所）
In this paper, we present a planner for a master dual-arm robot to manipulate tethered tools with an assistant dual-arm robot's help. The assistant robot provides assistance to the master robot by manipulating the tool cable and avoiding collisions. The provided assistance allows the master robot to perform tool placements on the robot workspace table to regrasp the tool, which would typically fail since the tool cable tension may change the tool positions. It also allows the master robot to perform tool handovers, which would normally cause entanglements or collisions with the cable and the e…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Deep reinforcement learning trains neural networks using experiences sampled from the replay buffer, which is commonly updated at each time step. In this paper, we propose a method to update the replay buffer adaptively and selectively to train a robot arm to accomplish a suction task in simulation. The response time of the agent is thoroughly taken into account. The state transitions that remain stuck at the boundary of constraint are not stored. The policy trained with our method works better than the one with the common replay buffer update method. The result is demonstrated both by simulat…
We present an interactive, computational design system for creating custom robotic arms given high-level task descriptions and environmental constraints. Various task requirements can be encoded as desired motion trajectories for the robot arm's end-effector. Given such end-effector trajectories, our system enables on-demand design of custom robot arms using a library of modular and reconfigurable parts such as actuators and connecting links. By searching through the combinatorial set of possible arrangements of these parts, our method generates a functional, as-simple-as-possible robot arm th…
In the scope of the "Chess-Bot" project, this study's goal is to choose the right model for the robotic arm that the "the Chess-Bot" will use to move the pawn from a cell to another. In this paper, there is the definition and the structure of a robot arm. Also, the different engineering and kinematics fundamentals of the robot and its components will be detailed. Furthermore, the different structures of robotic arms will be presented and compared based on different criteria. Finally, a model for "the Chess-Bot" arm will be synthesized based on accurate algorithms and equations.
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Soft robots promise improved safety and capability over rigid robots when deployed in complex, delicate, and dynamic environments. However, the infinite degrees of freedom and highly nonlinear dynamics of these systems severely complicate their modeling and control. As a step toward addressing this open challenge, we apply the data-driven, Hankel Dynamic Mode Decomposition (HDMD) with time delay observables to the model identification of a highly inertial, helical soft robotic arm with a high number of underactuated degrees of freedom. The resulting model is linear and hence amenable to contro…
Osaka University（大阪大学）；National Inst. of AIST（日本产业技术综合研究所）
This paper develops a planner that plans the action sequences and motion for a dual-arm robot to lift up and flip heavy plates using crane pulley blocks. The problem is motivated by the low payload of modern collaborative robots. Instead of directly manipulating heavy plates that collaborative robots cannot afford, the paper develops a planner for collaborative robots to operate crane pulley blocks. The planner assumes a target plate is pre-attached to the crane hook. It optimizes dual-arm action sequences and plans the robot's dual-arm motion that pulls the rope of the crane pulley blocks to …
Ubtech Robotics Corp.（优必选机器人公司）；Tsinghua University（清华大学）；Beijing Innovation Center for Future Chips, Tsinghua University（清华大学北京未来芯片创新中心）
When a dual-arm robot clamps a rigid object in an environment for human beings, the environment or the collaborating human will impose incidental disturbance on the operated object or the robot arm, leading to clamping failure, damaging the robot even hurting the human. This research proposes a prioritized hierarchical compliance control to simultaneously deal with the two types of disturbances in the dual-arm robot clamping. First, we use hierarchical quadratic programming (HQP) to solve the robot inverse kinematics under the joint constraints and prioritize the compliance for the disturbance…
In this paper, we present our work on developing robot arm prosthetic via deep learning. Our work proposes to use transfer learning techniques applied to the Google Inception model to retrain the final layer for surface electromyography (sEMG) classification. Data have been collected using the Thalmic Labs Myo Armband and used to generate graph images comprised of 8 subplots per image containing sEMG data captured from 40 data points per sensor, corresponding to the array of 8 sEMG sensors in the armband. Data captured were then classified into four categories (Fist, Thumbs Up, Open Hand, Rest…
The purpose of this research is to design, fabricate and test a tendon driven a continuum soft robot with three modular segments, each of which has a tunable stiffness enabled by layer jamming technology. Compared with previous studies, the robotic arm design of this project has a modular structure, which means the length of the robotic arm can be adjusted by addition of extra arm modules/segments to the existing robotic prototype. Furthermore, the new arm prototype supports motion within a 3-dimensional space. To achieve the goals, the design and fabrication for the variable stiffness robotic…
Manipulating articulated objects requires multiple robot arms in general. It is challenging to enable multiple robot arms to collaboratively complete manipulation tasks on articulated objects. In this paper, we present $\textbf{V-MAO}$, a framework for learning multi-arm manipulation of articulated objects. Our framework includes a variational generative model that learns contact point distribution over object rigid parts for each robot arm. The training signal is obtained from interaction with the simulation environment which is enabled by planning and a novel formulation of object-centric co…
We outline the design and construction of novel robotic arms using machine perception, convolutional neural networks, and symbolic AI for logical control and affordance indexing. We describe our robotic arms built with a humanlike mechanical configuration and aesthetic, with 28 degrees of freedom, touch sensors, and series elastic actuators. The arms were modelled in Roodle and Gazebo with URDF models, as well as Unity, and implement motion control solutions for solving live games of Baccarat (the casino card game), rock paper scissors, handshaking, and drawing. This includes live interactions…
The compliance of soft robotic arms renders the development of accurate kinematic & dynamical models especially challenging. The most widely used model in soft robotic kinematics assumes Piecewise Constant Curvature (PCC). However, PCC fails to effectively handle external forces, or even the influence of gravity, since the robot does not deform with a constant curvature under these conditions. In this paper, we establish three-dimensional (3D) modeling of a multi-segment soft robotic arm under the less restrictive assumption that each segment of the arm is deformed on a plane without twisting.…
Tecnalia France（法国泰克纳利亚）；LAAS-CNRS（法国国家科学研究中心自动化系统与机器人实验室）；University of Toulouse（图卢兹大学）；LIRMM（蒙彼利埃大学与法国国家科学研究中心联合机器人与微电子机械系统实验室）；Université de Montpellier（蒙彼利埃大学）；CNRS（法国国家科学研究中心）
A continuous collision checking method for a cable-driven parallel robot with an embarked robotic arm is proposed in this paper. The method aims at validating paths by checking for collisions between any pair of robot bodies (mobile platform, cables, and arm links). For a pair of bodies, an upper bound on their relative velocity and a lower bound on the distance between the bodies are computed and used to validate a portion of the path. These computations are done repeatedly until a collision is found or the path is validated. The method is integrated within the Humanoid Path Planner (HPP) sof…
Precision is a crucial performance indicator for robot arms, as high precision manipulation allows for a wider range of applications. Traditional methods for improving robot arm precision rely on error compensation. However, these methods are often not robust and lack adaptability. Learning-based methods offer greater flexibility and adaptability, while current researches show that they often fall short in achieving high precision and struggle to handle many scenarios requiring high precision. In this paper, we propose a novel high-precision robot arm manipulation framework based on online ite…
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Swansea University（斯旺西大学）
Industrial robot arms are extensively important for intelligent manufacturing. An industrial robot arm commonly enjoys its high repetitive positioning accuracy while suffering from its low absolute positioning accuracy, which greatly restricts its application in high-precision manufacture, like automobile manufacture. Aiming at addressing this hot issue, this work proposes a novel robot arm calibration method based on cubic interpolated beetle antennae search (CIBAS). This study has three ideas: a) developing a novel CIBAS algorithm, which can effectively addresses the issue of local optimum i…
This paper investigates the problem of the dynamic balance control of multi-arm free-floating space robot during capturing an active object in close proximity. The position and orientation of space base will be affected during the operation of space manipulator because of the dynamics coupling between the manipulator and space base. This dynamics coupling is unique characteristics of space robot system. Such a disturbance will produce a serious impact between the manipulator hand and the object. To ensure reliable and precise operation, we propose to develop a space robot system consisting of …
Generating and Customizing Robotic Arm Trajectories using Neural Networks
Faculty of Mathematics, Physics and Informatics（数学、物理与信息学系）；Comenius University Bratislava（布拉迪斯拉发康门尼大学）；DIBRIS（迪布里斯）；University of Genoa（热那亚大学）；Italian Institute of Technology（意大利理工学院）
Robotic deformable-object manipulation is a challenge in the robotic industry because deformable objects have complicated and various object states. Predicting those object states and updating manipulation planning is time-consuming and computationally expensive. In this paper, we propose learning known configurations of garments to allow a robot to recognise garment states and choose a pre-designed manipulation plan for garment flattening.
University of Bordeaux（波尔多大学）；University Politehnica of Bucharest（布加勒斯特理工大学）
Attempts to install a rotating tool at the end of a robot arm poly-articulated date back twenty years, but these robots were not designed for that. Indeed, two essential features are necessary for machining: high rigidity and precision in a given workspace. The experimental results presented are the dynamic identification of a poly-articulated robot equipped with an integrated spindle. This study aims to highlight the influence of the geometric configuration of the robot arm on the overall stiffness of the system. The spindle is taken into account as an additional weight on board but also as a…
King’s College London（伦敦国王学院）；Xtronics Ltd.（Xtronics有限公司）；Guy’s & St Thomas’ NHS Foundation Trust（盖伊和圣托马斯NHS基金会信托）；Evelina Children’s Hospital（伊芙琳娜儿童医院）；Queen Mary University of London（伦敦玛丽女王大学）
The development of robotic-assisted extracorporeal ultrasound systems has a long history and a number of projects have been proposed since the 1990s focusing on different technical aspects. These aim to resolve the deficiencies of on-site manual manipulation of hand-held ultrasound probes. This paper presents the recent ongoing developments of a series of bespoke robotic systems, including both single-arm and dual-arm versions, for a project known as intelligent Fetal Imaging and Diagnosis (iFIND). After a brief review of the development history of the extracorporeal ultrasound robotic system …
Teleoperation provides a way for human operators to guide robots in situations where full autonomy is challenging or where direct human intervention is required. It can also be an important tool to teach robots in order to achieve autonomous behaviour later on. The increased availability of collaborative robot arms and Virtual Reality (VR) devices provides ample opportunity for development of novel teleoperation methods. Since robot arms are often kinematically different from human arms, mapping human motions to a robot in real-time is not trivial. Additionally, a human operator might steer th…
Using results on the topology of moduli space of polygons [Jaggi, 92; Kapovich and Millson, 94], it can be shown that for a planar robot arm with $n$ segments there are some values of the base-length, $z$, at which the configuration space of the constrained arm (arm with its end effector fixed) has two disconnected components, while at other values the constrained configuration space has one connected component. We first review some of these known results. Then the main design problem addressed in this paper is the construction of pairs of continuous inverse kinematics for arbitrary robot arms…
Examining the legibility of humanoid robot arm movements in a pointing task
Faculty of Mathematics, Physics；Informatics, Comenius University Bratislava, Slovakia DIBRIS, University of Genoa, Genoa, Italy CONTACT, Italian Institute of Technology, Genoa, Italy Digital Health Dept., Nvision Systems \& Technologies, S.L., Barcelona, Spain Faculty of Education, University of Ljubljana, Slovenia
Recent trends in robot arm control have seen a shift towards end-to-end solutions, using deep reinforcement learning to learn a controller directly from raw sensor data, rather than relying on a hand-crafted, modular pipeline. However, the high dimensionality of the state space often means that it is impractical to generate sufficient training data with real-world experiments. As an alternative solution, we propose to learn a robot controller in simulation, with the potential of then transferring this to a real robot. Building upon the recent success of deep Q-networks, we present an approach …
