We designed a modular robotic control stack that provides a customizable and accessible interface to the Franka Emika Panda Research robot. This framework abstracts high-level robot control commands as skills, which are decomposed into combinations of trajectory generators, feedback controllers, and termination handlers. Low-level control is implemented in C++ and runs at $1$kHz, and high-level commands are exposed in Python. In addition, external sensor feedback, like estimated object poses, can be streamed to the low-level controllers in real time. This modular approach allows us to quickly prototype new control methods, which is essential for research applications. We have applied this framework across a variety of real-world robot tasks in more than $5$ published research papers. The framework is currently shared internally with other robotics labs at Carnegie Mellon University, and we plan for a public release in the near future.
Istituto Italiano di Tecnologia（意大利理工学院）；Universita di Pisa（比萨大学）
Teleoperation is a widely adopted strategy to control robotic manipulators executing complex tasks that require highly dexterous movements and critical high-level intelligence. Classical teleoperation schemes are based on either joystick control, or on more intuitive interfaces which map directly the user arm motions into one robot arm's motions. These approaches have limits when the execution of a given task requires reconfigurable multiple robotic arm systems. Indeed, the simultaneous teleoperation of two or more robot arms could extend the workspace of the manipulation cell, or increase its…
Stack-of-Tasks (SoT) control allows a robot to simultaneously fulfill a number of prioritized goals formulated in terms of (in)equality constraints in error space. Since this approach solves a sequence of Quadratic Programs (QP) at each time-step, without taking into account any temporal state evolution, it is suitable for dealing with local disturbances. However, its limitation lies in the handling of situations that require non-quadratic objectives to achieve a specific goal, as well as situations where countering the control disturbance would require a locally suboptimal action. Recent work…
A Novel Modular Cable-Driven Soft Robotic Arm with Multi-Segment Reconfigurability
Department of Agricultural and Biological Engineering, Mississippi State University（农业与生物工程系，密苏里州立大学）；Department of Electrical and Computer Engineering, Michigan State University（电气与计算机工程系，密苏里州立大学）；Texas A&M AgriLife Research and Extension Center in Dallas, Texas A&M University（德克萨斯A&M大学达拉斯分校农业与生命科学研究中心）
BaRiFlex: A Robotic Gripper with Versatility and Collision Robustness for Robot Learning
Department of Mechanical Engineering, The University of Texas at Austin（机械工程系，德克萨斯大学奥斯汀分校）；Department of Computer Science, The University of Texas at Austin（计算机科学系，德克萨斯大学奥斯汀分校）
University of Tübingen（蒂宾根大学）；Graz University of Technology（格拉茨技术大学）
In the paper, we show how scalable, low-cost trunk-like robotic arms can be constructed using only basic 3D-printing equipment and simple electronics. The design is based on uniform, stackable joint modules with three degrees of freedom each. Moreover, we present an approach for controlling these robots with recurrent spiking neural networks. At first, a spiking forward model learns motor-pose correlations from movement observations. After training, intentions can be projected back through unrolled spike trains of the forward model essentially routing the intention-driven motor gradients towar…
Delft University of Technology（代尔夫特理工大学）；Radboud University（拉德堡德大学）
Adaptation to external and internal changes is major for robotic systems in uncertain environments. Here we present a novel multisensory active inference torque controller for industrial arms that shows how prediction can be used to resolve adaptation. Our controller, inspired by the predictive brain hypothesis, improves the capabilities of current active inference approaches by incorporating learning and multimodal integration of low and high-dimensional sensor inputs (e.g., raw images) while simplifying the architecture. We performed a systematic evaluation of our model on a 7DoF Franka Emik…
Delft University of Technology（代尔夫特理工大学）；Radboud University（拉德堡德大学）；Donders Institute for Brain, Cognition and Behavior（唐德斯脑、认知与行为研究所）
Active inference, a theoretical construct inspired by brain processing, is a promising alternative to control artificial agents. However, current methods do not yet scale to high-dimensional inputs in continuous control. Here we present a novel active inference torque controller for industrial arms that maintains the adaptive characteristics of previous proprioceptive approaches but also enables large-scale multimodal integration (e.g., raw images). We extended our previous mathematical formulation by including multimodal state representation learning using a linearly coupled multimodal variat…
Goal-oriented Semantic Communication for Robot Arm Reconstruction in Digital Twin: Feature and Temporal Selections
Department of Engineering, King’s College London（伦敦国王学院工程系）；ZhejiangLab（浙江实验室）；Bristol Research and Innovation Laboratory, Toshiba Europe Ltd.（巴斯德研究与创新实验室，东芝欧洲有限公司）
In this paper, we present a framework that unites obstacle avoidance and deliberate physical interaction for robotic manipulators. As humans and robots begin to coexist in work and household environments, pure collision avoidance is insufficient, as human-robot contact is inevitable and, in some situations, desired. Our work enables manipulators to anticipate, detect, and act on contact. To achieve this, we allow limited deviation from the robot's original trajectory through velocity reduction and motion restrictions. Then, if contact occurs, a robot can detect it and maneuver based on a novel…
In this work we propose a learning approach to high-precision robotic assembly problems. We focus on the contact-rich phase, where the assembly pieces are in close contact with each other. Unlike many learning-based approaches that heavily rely on vision or spatial tracking, our approach takes force/torque in task space as the only observation. Our training environment is robotless, as the end-effector is not attached to any specific robot. Trained policies can then be applied to different robotic arms without re-training. This approach can greatly reduce complexity to perform contact-rich rob…
Istituto Italiano di Tecnologia（意大利理工学院）；University of Naples Federico II（那不勒斯费德里科二世大学）
Enabling robots to work in close proximity to humans necessitates a control framework that does not only incorporate multi-sensory information for autonomous and coordinated interactions but also has perceptive task planning to ensure an adaptable and flexible collaborative behaviour. In this research, an intuitive stack-of-tasks (iSoT) formulation is proposed, that defines the robot's actions by considering the human-arm postures and the task progression. The framework is augmented with visuo-tactile information to effectively perceive the collaborative environment and intuitively switch betw…
Robotic manipulation of cloth has applications ranging from fabrics manufacturing to handling blankets and laundry. Cloth manipulation is challenging for robots largely due to their high degrees of freedom, complex dynamics, and severe self-occlusions when in folded or crumpled configurations. Prior work on robotic manipulation of cloth relies primarily on vision sensors alone, which may pose challenges for fine-grained manipulation tasks such as grasping a desired number of cloth layers from a stack of cloth. In this paper, we propose to use tactile sensing for cloth manipulation; we attach a…
Graph-Fused Vision-Language-Action for Policy Reasoning in Multi-Arm Robotic Manipulation
Machine Learning and I-health International Cooperation Base of Zhejiang Province, Artificial Intelligence Institute, Hangzhou Dianzi University, Zhejiang（浙江省机器学习与健康国际合作基地、人工智能学院、杭州电子大学）；Electrical and Computer Engineering Department, The University of New Mexico（新墨西哥大学电气与计算机工程系）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）
Aalto University（阿尔托大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
Training end-to-end deep robot policies requires a lot of domain-, task-, and hardware-specific data, which is often costly to provide. In this work, we propose to tackle this issue by employing a deep neural network with a modular architecture, consisting of separate perception, policy, and trajectory parts. Each part of the system is trained fully on synthetic data or in simulation. The data is exchanged between parts of the system as low-dimensional latent representations of affordances and trajectories. The performance is then evaluated in a zero-shot transfer scenario using Franka Panda r…
Multi-task learning ideally allows robots to acquire a diverse repertoire of useful skills. However, many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In reality, the tasks that the robot learns arrive sequentially, depending on the user and the robot's current environment. In this work, we study a practical sequential multi-task RL problem that is motivated by the practical constraints of physical robotic systems, and derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow t…
CSymPlan: Certified Symbolic Planning and Control for High-DOF Manipulators
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学（MBZUAI））；Technical University of Munich（慕尼黑工业大学）；Ludwig Maximilian University of Munich（慕尼黑大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）
Learning-Based Robust Control: Unifying Exploration and Distributional Robustness for Reliable Robotics via Free Energy
Mechanical Engineering Program, New York University Abu Dhabi（纽约大学阿布扎赫尔分校机械工程项目）；Department of Information and Electrical Engineering & Applied Mathematics, University of Salerno（萨勒诺大学信息与电气工程及应用数学系）；Mohammed bin Zayed University for Artificial Intelligence, Abu Dhabi, UAE（穆罕默德·本·扎耶德人工智能大学）
Standardized evaluation measures have aided in the progress of machine learning approaches in disciplines such as computer vision and machine translation. In this paper, we make the case that robotic learning would also benefit from benchmarking, and present the "REPLAB" platform for benchmarking vision-based manipulation tasks. REPLAB is a reproducible and self-contained hardware stack (robot arm, camera, and workspace) that costs about 2000 USD, occupies a cuboid of size 70x40x60 cm, and permits full assembly within a few hours. Through this low-cost, compact design, REPLAB aims to drive wid…
Long horizon sequential manipulation tasks are effectively addressed hierarchically: at a high level of abstraction the planner searches over abstract action sequences, and when a plan is found, lower level motion plans are generated. Such a strategy hinges on the ability to reliably predict that a feasible low level plan will be found which satisfies the abstract plan. However, computing Abstract Plan Feasibility (APF) is difficult because the outcome of a plan depends on real-world phenomena that are difficult to model, such as noise in estimation and execution. In this work, we present an a…
We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA falls under the shared autonomy paradigm: in addition to providing discrete language inputs, humans are given a low-dimensional controller $-$ e.g., a 2 degree-of-freedom (DoF) joystick that can move left/right and up/down $-$ for operating the robot. LILA learns to use language to modulate this controller, providing users with a language-informed control space: given an instruction like "place the cereal bowl on the tray," LILA may learn …
ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulation
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人学院）；IIIS, Tsinghua University（清华大学人工智能研究所）；Department of Computer Science, National Yang Ming Chiao Tung University（国立阳明交通大学计算机科学系）
Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）；University of California, Berkeley（加州大学伯克利分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Learning Stack-of-Tasks Management for Redundant Robots
University of Padova, Dept. of Information Engineering（帕多瓦大学信息工程系）；ABB Corporate Research（ABB企业研究）；Mobile Robotics Lab, ETH Zürich（苏黎世联邦理工学院移动机器人实验室）；Polytechnic of Bari Dept. of Electrical and Information Engineering（巴里理工学院电气与信息工程系）
