This paper addresses a new strategy called Simulation-to-Real-to-Simulation (Sim2Real2Sim) to bridge the gap between simulation and real-world, and automate a flexible object manipulation task. This strategy consists of three steps: (1) using the rough environment with the estimated models to develop the methods to complete the manipulation task in the simulation; (2) applying the methods from simulation to real-world and comparing their performance; (3) updating the models and methods in simulation based on the differences between the real world and the simulation. The Plug Task from the 2015 DARPA Robotics Challenge Finals is chosen to evaluate our Sim2Real2Sim strategy. A new identification approach for building the model of the linear flexible objects is derived from real-world to simulation. The automation of the DRC plug task in both simulation and real-world proves the success of the Sim2Real2Sim strategy. Numerical experiments are implemented to validate the simulated model.
Reflective VLM Planning for Dual-Arm Desktop Cleaning: Bridging Open-Vocabulary Perception and Precise Manipulation
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Department of Electrical and Computer Engineering, Carnegie Mellon University（卡内基梅隆大学电气与计算机工程系）；College of Aeronautics & Engineering, Kent State University（肯特州立大学航空与工程学院）
A neural network based flexible object manipulation system for a humanoid robot on FPGA is proposed. Although the manipulations of flexible objects using robots attract ever increasing attention since these tasks are the basic and essential activities in our daily life, it has been put into practice only recently with the help of deep neural networks. However such systems have relied on GPU accelerators, which cannot be implemented into the space limited robotic body. Although field programmable gate arrays (FPGAs) are known to be energy efficient and suitable for embedded systems, the model s…
DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation
DexRobot Co. Ltd.（德克斯机器人有限公司）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；Chongqing University（重庆大学）；East China University of Science and Technology（东华大学）
Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
Technical University of Munich（慕尼黑技术大学）；Corporate Research, Robert Bosch GmbH（罗伯特·博世集团企业研究部）；University of California Berkeley（加州大学伯克利分校）；Microsoft（微软）；Google DeepMind（谷歌DeepMind）；Carnegie Mellon University（卡内基梅隆大学）；Imperial College London（伦敦帝国理工学院）；Princeton University（普林斯顿大学）；University of Southern California（南加州大学）；Sun Yat-sen University（中山大学）；University of Michigan（密歇根大学）；Institute for Artificial Intelligence, University of Stuttgart（斯图加特大学人工智能研究所）；The State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
Manipulation of deformable objects is a desired skill in making robots ubiquitous in manufacturing, service, healthcare, and security. Deformable objects are common in our daily lives, e.g., wires, clothes, bed sheets, etc., and are significantly more difficult to model than rigid objects. In this study, we investigate vision-based manipulation of linear flexible objects such as cables. We propose a geometric modeling method that is based on visual feedback to develop a general representation of the linear flexible object that is subject to gravity. The model characterizes the shape of the obj…
We achieved contact-rich flexible object manipulation, which was difficult to control with vision alone. In the unzipping task we chose as a validation task, the gripper grasps the puller, which hides the bag state such as the direction and amount of deformation behind it, making it difficult to obtain information to perform the task by vision alone. Additionally, the flexible fabric bag state constantly changes during operation, so the robot needs to dynamically respond to the change. However, the appropriate robot behavior for all bag states is difficult to prepare in advance. To solve this …
Learning to solve precision-based manipulation tasks from visual feedback using Reinforcement Learning (RL) could drastically reduce the engineering efforts required by traditional robot systems. However, performing fine-grained motor control from visual inputs alone is challenging, especially with a static third-person camera as often used in previous work. We propose a setting for robotic manipulation in which the agent receives visual feedback from both a third-person camera and an egocentric camera mounted on the robot's wrist. While the third-person camera is static, the egocentric camera…
We present a method for system identification of flexible objects by measuring forces and displacement during interaction with a manipulating arm. We model the object's structure and flexibility by a chain of rigid bodies connected by torsional springs. Unlike previous work, the proposed optimal control approach using variational integrators allows identification of closed loops, which include the robot arm itself. This allows using the resulting models for planning in configuration space of the robot. In order to solve the resulting problem efficiently, we develop a novel method for fast disc…
University of Genoa（热那亚大学）；University of Huddersfield（哈德斯菲尔德大学）
Articulated and flexible objects constitute a challenge for robot manipulation tasks but are present in different real-world settings, including home and industrial environments. Current approaches to the manipulation of articulated and flexible objects employ ad hoc strategies to sequence and perform actions on them depending on a number of physical or geometrical characteristics related to those objects, as well as on an a priori classification of target object configurations. In this paper, we propose an action planning and execution framework, which (i) considers abstract representations o…
MIT Lincoln Laboratory（麻省理工学院林肯实验室）；Systems and Technology Research（系统与技术研究公司）
Significant progress has been made in scene understanding which seeks to build 3D, metric and object-oriented representations of the world. Concurrently, reinforcement learning has made impressive strides largely enabled by advances in simulation. Comparatively, there has been less focus in simulation for perception algorithms. Simulation is becoming increasingly vital as sophisticated perception approaches such as metric-semantic mapping or 3D dynamic scene graph generation require precise 3D, 2D, and inertial information in an interactive environment. To that end, we present TESSE (Task Exec…
Reinforcement learning (RL) has been shown to be effective at learning control from experience. However, RL typically requires a large amount of online interaction with the environment. This limits its applicability to real-world settings, such as in robotics, where such interaction is expensive. In this work we investigate ways to minimize online interactions in a target task, by reusing a suboptimal policy we might have access to, for example from training on related prior tasks, or in simulation. To this end, we develop two RL algorithms that can speed up training by using not only the acti…
Closed kinematic chains are created whenever multiple robot arms concurrently manipulate a single object. The closed-chain constraint, when coupled with robot joint limits, dramatically changes the connectivity of the configuration space. We propose a regrasping move, termed "IK-switch", which allows efficiently bridging components of the configuration space that are otherwise mutually disconnected. This move, combined with several other developments, such as a method to stabilize the manipulated object using the environment, a new tree structure, and a compliant control scheme, enables us to …
BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fields
School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）；Key Laboratory of Advanced Manufacturing Technology of the Ministry of Education, Guizhou University（教育部贵州大学先进制造技术重点实验室）
Humans demonstrate a variety of interesting behavioral characteristics when performing tasks, such as selecting between seemingly equivalent optimal actions, performing recovery actions when deviating from the optimal trajectory, or moderating actions in response to sensed risks. However, imitation learning, which attempts to teach robots to perform these same tasks from observations of human demonstrations, often fails to capture such behavior. Specifically, commonly used learning algorithms embody inherent contradictions between the learning assumptions (e.g., single optimal action) and actu…
Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos
The University of Hong Kong（香港大学）；ARC Lab, Tencent PCG（腾讯PCG实验室）；The Chinese University of Hong Kong（香港中文大学）；University of California, Berkeley（加州大学伯克利分校）
Vibration Suppression in Collaborative Flexible Payload Manipulation Using Passive Force Control
VTT Technical Research Centre of Finland Ltd（芬兰VTT技术研究中心）；RP Technical University of Kaiserslautern（凯泽斯劳滕RP工业大学）；UK Atomic Energy Authority（英国原子能管理局）；University of Oulu（奥卢大学）
Introducing Environmental Constraints to Grasping Strategies for Paper-Like Flexible Materials Using a Soft Gripper
College of Mechanical and Electrical Engineering, Nanjing University（南京大学机械与电子工程学院）；Jiangsu Key Laboratory of Bionic Materials and Equipment, Nanjing University of Aeronautics and Astronautics（江苏生物材料与设备重点实验室，南京航空航天大学）
Integration of a Variable Stiffness Link for Long-Reach Aerial Manipulation
Department of Electrical and Photonics Engineering Control, Robotics and Embodied AI, Technical University of Denmark（技术大学丹麦分校电气与光子工程系）；GRVC Robotics Laboratory, Universidad de Sevilla（塞维利亚大学GRVC机器人实验室）
EnvBridge: Bridging Diverse Environments with Cross-Environment Knowledge Transfer for Embodied AI
Panasonic Connect Co., Ltd.（松下连接有限公司）；National University of Singapore（新加坡国立大学）；Panasonic R&D Center Singapore（松下新加坡研发中心）；Panasonic R&D Center Germany（松下德国研发中心）
Manipulation of Elasto-Flexible Cables with Single or Multiple UAVs
Robotics and Mechatronics Department, Electrical Engineering, Mathematics, and Computer Science (EEMCS) Faculty, University of Twente（代尔夫特理工大学机器人与机电学系）；Department of Computer, Control and Management Engineering, Sapienza University of Rome（罗马萨皮恩扎大学计算机、控制与管理工程系）
