This work introduces our approach to the flat and textureless object grasping problem. In particular, we address the tableware and cutlery manipulation problem where a service robot has to clean up a table. Our solution integrates colour and 2D and 3D geometry information to describe objects, and this information is given to the robot action planner to find the best grasping trajectory depending on the object class. Furthermore, we use visual feedback as a verification step to determine if the grasping process has successfully occurred. We evaluate our approach in both an open and a standard service robot platform following the RoboCup@Home international tournament regulations.
We present a passive stereo depth system that produces dense and accurate point clouds optimized for human environments, including dark, textureless, thin, reflective and specular surfaces and objects, at 2560x2048 resolution, with 384 disparities, in 30 ms. The system consists of an algorithm combining learned stereo matching with engineered filtering, a training and data-mixing methodology, and a sensor hardware design. Our architecture is 15x faster than approaches that perform similarly on the Middlebury and Flying Things Stereo Benchmarks. To effectively supervise the training of this mod…
Given the laborious difficulty of moving heavy bags of physical currency in the cash center of the bank, there is a large demand for training and deploying safe autonomous systems capable of conducting such tasks in a collaborative workspace. However, the deformable properties of the bag along with the large quantity of rigid-body coins contained within it, significantly increases the challenges of bag detection, grasping and manipulation by a robotic gripper and arm. In this paper, we apply deep reinforcement learning and machine learning techniques to the task of controlling a collaborative …
By learning Variable Impedance Control policy, robot assistants can intelligently adapt their manipulation compliance to ensure both safe interaction and proper task completion when operating in human-robot interaction environments. In this paper, we propose a DMP-based framework that learns and generalizes variable impedance manipulation skills from human demonstrations. This framework improves robots$'$ adaptability to environment changes(i.e. the weight and shape changes of grasping object at the robot end-effector) and inherits the efficiency of demonstration-variance-based stiffness estim…
Carnegie Mellon University（卡内基梅隆大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Brown University（布朗大学）
A key challenge in intelligent robotics is creating robots that are capable of directly interacting with the world around them to achieve their goals. The last decade has seen substantial growth in research on the problem of robot manipulation, which aims to exploit the increasing availability of affordable robot arms and grippers to create robots capable of directly interacting with the world to achieve their goals. Learning will be central to such autonomous systems, as the real world contains too much variation for a robot to expect to have an accurate model of its environment, the objects …
Do Robots Really Need Anthropomorphic Hands? A Comparison of Human and Robotic Hands
Robotics Innovation Center, Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)（德意志人工智能研究中心机器人创新中心）；Leibniz Universität Hannover, L3S Research Center（汉诺威莱布尼茨大学L3S研究中心）
In this work, we present a manipulation planning algorithm for a robot to keep an object stable under changing external forces. We particularly focus on the case where a human may be applying forceful operations, e.g. cutting or drilling, on an object that the robot is holding. The planner produces an efficient plan by intelligently deciding when the robot should change its grasp on the object as the human applies the forces. The planner also tries to choose subsequent grasps such that they will minimize the number of regrasps that will be required in the long-term. Furthermore, as it switches…
Manipulation tasks in daily life, such as pouring water, unfold intentionally under specialized manipulation contexts. Being able to process contextual knowledge in these Activities of Daily Living (ADLs) over time can help us understand manipulation intentions, which are essential for an intelligent robot to transition smoothly between various manipulation actions. In this paper, to model the intended concepts of manipulation, we present a vision dataset under a strictly constrained knowledge domain for both robot and human manipulations, where manipulation concepts and relations are stored b…
Diffusion Stabilizer Policy for Automated Surgical Robot Manipulations
UM-SJTU Joint Intuitute（UM-SJTU联合学院）；Shanghai Jiao Tong University（上海交通大学）；Department of Computer Science and Engineering（计算机科学与工程系）；The Chinese University of Hong Kong（香港中文大学）；Department of Automation（自动化系）
University of Bremen（不来梅大学）；Technical University of Munich（慕尼黑工业大学）；Fyuse Inc.（Fyuse公司）；German Aerospace Center (DLR)（德国航空航天中心（DLR））
A pressing question when designing intelligent autonomous systems is how to integrate the various subsystems concerned with complementary tasks. More specifically, robotic vision must provide task-relevant information about the environment and the objects in it to various planning related modules. In most implementations of the traditional Perception-Cognition-Action paradigm these tasks are treated as quasi-independent modules that function as black boxes for each other. It is our view that perception can benefit tremendously from a tight collaboration with cognition. We present RoboSherlock,…
Energy-Aware Reinforcement Learning for Robotic Manipulation of Articulated Components in Infrastructure Operation and Maintenance
National Key Laboratory of Automotive Chassis Integration and Bionics（汽车底盘集成与仿生国家重点实验室）；Jilin University（吉林大学）；School of Computer Science and Statistics（计算机科学与统计学学院）；Trinity College Dublin（都柏林大学）；Department of Civil Engineering（土木工程系）
Learning actions from human demonstration is an emerging trend for designing intelligent robotic systems, which can be referred as video to command. The performance of such approach highly relies on the quality of video captioning. However, the general video captioning methods focus more on the understanding of the full frame, lacking of consideration on the specific object of interests in robotic manipulations. We propose a novel deep model to learn actions from human demonstration video for robotic manipulation. It consists of two deep networks, grasp detection network (GNet) and video capti…
Generalizable object manipulation skills are critical for intelligent and multi-functional robots to work in real-world complex scenes. Despite the recent progress in reinforcement learning, it is still very challenging to learn a generalizable manipulation policy that can handle a category of geometrically diverse articulated objects. In this work, we tackle this category-level object manipulation policy learning problem via imitation learning in a task-agnostic manner, where we assume no handcrafted dense rewards but only a terminal reward. Given this novel and challenging generalizable poli…
University of Utah Robotics Center（犹他大学机器人中心）；NVIDIA Corporation（英伟达公司）
Ergonomics and human comfort are essential concerns in physical human-robot interaction. Common practical methods in the area either fail in estimating the correct posture due to occlusion or suffer from inaccurate ergonomics models in performing postural optimization. We propose a novel alternative framework for posture estimation, assessment, and optimization for ergonomically intelligent physical human-robot interaction. We show that we can estimate human posture solely from the trajectory of the interacting robot with median deviation of 5 deg from motion capture. We propose DULA, a differ…
We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closed-loop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable obj…
Stanford University（斯坦福大学）；NVIDIA（英伟达）；California Institute of Technology（加州理工学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Learning performant robot manipulation policies can be challenging due to high-dimensional continuous actions and complex physics-based dynamics. This can be alleviated through intelligent choice of action space. Operational Space Control (OSC) has been used as an effective task-space controller for manipulation. Nonetheless, its strength depends on the underlying modeling fidelity, and is prone to failure when there are modeling errors. In this work, we propose OSC for Adaptation and Robustness (OSCAR), a data-driven variant of OSC that compensates for modeling errors by inferring relevant dy…
This thesis is concerned with deriving planning algorithms for robot manipulators. Manipulation has two effects, the robot has a physical effect on the object, and it also acquires information about the object. This thesis presents algorithms that treat both problems. First, I present an extension of the well-known piano mover's problem where a robot pushing an object must plan its movements as well as those of the object. This requires simultaneous planning in the joint space of the robot and the configuration space of the object, in contrast to the original problem which only requires planni…
The logistics of transporting a package from a storage facility to the consumer's front door usually employs highly specialized robots often times splitting sub-tasks up to different systems, e.g., manipulator arms to sort and wheeled vehicles to deliver. More recent endeavors attempt to have a unified approach with legged and humanoid robots. These solutions, however, occupy large amounts of space thus reducing the number of packages that can fit into a delivery vehicle. As a result, these bulky robotic systems often reduce the potential for scalability and task parallelization. In this paper…
Simulation of Optical Tactile Sensors Supporting Slip and Rotation using Path Tracing and IMPM
Zhili College, Tsinghua University（清华大学知礼学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Engineering and Technology, China University of Geosciences (Beijing)（中国地质大学（北京）工程技术学院）
University of Lincoln（林肯大学）；Lincoln Institute for Agri-Food Technology（林肯农业食品技术研究所）
Robots learning a new manipulation task from a small amount of demonstrations are increasingly demanded in different workspaces. A classifier model assessing the quality of actions can predict the successful completion of a task, which can be used by intelligent agents for action-selection. This paper presents a novel classifier that learns to classify task completion only from a few demonstrations. We carry out a comprehensive comparison of different neural classifiers, e.g. fully connected-based, fully convolutional-based, sequence2sequence-based, and domain adaptation-based classification. …
SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining
Institute of Artificial Intelligence, China Telecom（中国电信人工智能研究院）；Technical University of Munich（慕尼黑工业大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）
Autonomous Manipulation of Hazardous Chemicals and Delicate Objects in a Self-Driving Laboratory: A Sliding Mode Approach
Department of Electronic Systems, Aalborg University（电子系统系，奥尔堡大学）；Department of Information Technology and Electrical Engineering, University of Naples（信息科技与电气工程系，那不勒斯大学）
DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulation
Institute for Artificial Intelligence（人工智能研究院）；School of Integrated Circuits, Peking University（集成电路学院，北京大学）；Shenzhen Institute of Artificial Intelligence（深圳人工智能研究所）；School of Electronics Engineering and Computer Science, Peking University（电子工程与计算机科学学院，北京大学）；Beijing Advanced Innovation Center for Integrated Circuits（北京集成电路先进制造创新中心）
An Integrated System for WEEE Sorting Employing X-ray Imaging, AI-based Object Detection and Segmentation, and Delta Robot Manipulation
THL (Tech Hive Labs)（Tech Hive Labs）；AI Innovation Centre, University of Essex（艾创新中心，埃塞克斯大学）；Control Systems Lab, School of Mechanical Engineering, NTUA（控制系统实验室，机械工程学院，NTUA）
Whole-Body Proprioceptive Morphing: A Modular Soft Gripper for Robust Cross-Scale Grasping
Mechanical Engineering Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机械工程系）；Robotics Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机器人学系）
Multi-Sensor Fusion-Based Mobile Manipulator Remote Control for Intelligent Smart Home Assistance
Dyson School of Design Engineering, Imperial College London（帝国理工学院伦敦设计工程学院）；School of Robotics, Xi’an Jiaotong-Liverpool University (XJTLU)（西安交通大学利物浦大学机器人学院）；Department of Engineering, University of Cambridge（剑桥大学工程学院）；School of Mechanical Engineering, Northwestern Polytechnical University（西北工业大学机械工程学院）；Department of Computer Science, University of Aberdeen（阿伯丁大学计算机科学学院）
Interactive Learning of Physical Object Properties Through Robot Manipulation and Database of Object Measurements
Czech Technical University in Prague（布拉格捷克技术大学）；Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague（布拉格捷克技术大学捷克信息学、机器人学与控制论研究所）；New York University Abu Dhabi（纽约大学阿布扎比分校）；Mondragon Unibertsitatea（蒙德拉贡大学）；Imperial College London（伦敦帝国学院）；Aalto University（阿尔托大学）
Object surface reconstruction brings essential benefits to robot grasping, object recognition, and object manipulation. When measuring the surface distribution of an unknown object by tapping, the greatest challenge is to select tapping positions efficiently and accurately without prior knowledge of object region. Given a searching range, we propose an active exploration method, to efficiently and intelligently guide the tapping to learn the object surface without exhaustive and unnecessary off-surface tapping. We analyze the performance of our approach in modeling object surfaces within an ex…
Latest research in industrial robotics is aimed at making human robot collaboration possible seamlessly. For this purpose, industrial robots are expected to work on the fly in unstructured and cluttered environments and hence the subject of perception driven motion planning plays a vital role. Sampling based motion planners are proven to be the most effective for such high dimensional planning problems with real time constraints. Unluckily random stochastic samplers suffer from the phenomenon of 'narrow passages' or bottleneck regions which need targeted sampling to improve their convergence r…
Center of Intelligent Robotics, Indian Institute of Information Technology, Allahabad（印度阿拉哈巴德信息技术学院智能机器人中心）；Indian Institute of Technology (Indian School of Mines), Dhanbad（印度丹巴德印度理工学院（印度矿业学院））
Intelligent Object manipulation for grasping is a challenging problem for robots. Unlike robots, humans almost immediately know how to manipulate objects for grasping due to learning over the years. A grown woman can grasp objects more skilfully than a child because of learning skills developed over years, the absence of which in the present day robotic grasping compels it to perform well below the human object grasping benchmarks. In this paper we have taken up the challenge of developing learning based pose estimation by decomposing the problem into both position and orientation learning. Mo…
KTH Royal Institute of Technology（KTH皇家理工学院）；School of Electrical Engineering and Computer Science（电气工程与计算机科学学院）
Control and planning of multi-agent systems is an active and increasingly studied topic of research, with many practical applications such as rescue missions, security, surveillance, and transportation. This thesis addresses the planning and control of multi-agent systems under temporal logic tasks. The considered systems concern complex, robotic, manipulator-endowed systems, which can coordinate in order to execute complicated tasks, including object manipulation/transportation. Motivated by real-life scenarios, we take into account high-order dynamics subject to model uncertainties and unkno…
KTH Stockholm（斯德哥尔摩皇家理工学院）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；MILA（米拉人工智能研究所）
Dexterous manipulation remains an open problem in robotics. To coordinate efforts of the research community towards tackling this problem, we propose a shared benchmark. We designed and built robotic platforms that are hosted at MPI for Intelligent Systems and can be accessed remotely. Each platform consists of three robotic fingers that are capable of dexterous object manipulation. Users are able to control the platforms remotely by submitting code that is executed automatically, akin to a computational cluster. Using this setup, i) we host robotics competitions, where teams from anywhere in …
A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One way of such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable to establish a common focus of attention and be able to use and integrate spoken instructions, visual perceptions, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks…
