The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states (i.e.\ static object poses and fingertip locations) for various in-hand manipulation tasks. We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across all tasks, also serves as a measure of the system's in-hand manipulation capability. We provide supporting software, task examples, and evaluation results associated with the benchmark. All the supporting material is available at https://robot-learning.cs.utah.edu/project/benchmarking_in_hand_manipulation
Developing Modular Grasping and Manipulation Pipeline Infrastructure to Streamline Performance Benchmarking
New England Robotics Validation and Experimentation (NERVE) Center, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；Rutgers University（罗格斯大学）；Worcester Polytechnic University（伍斯特理工大学）；Yale University（耶鲁大学）；University of South Florida（南佛罗里达大学）
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息学研究所）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Benchmarking of robotic manipulations is one of the open issues in robotic research. An important factor that has enabled progress in this area in the last decade is the existence of common object sets that have been shared among different research groups. However, the existing object sets are very limited when it comes to cloth-like objects that have unique particularities and challenges. This paper is a first step towards the design of a cloth object set to be distributed among research groups from the robotics cloth manipulation community. We present a set of household cloth objects and rel…
Data61, Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO）Data61研究所）；Queensland University of Technology (QUT)（昆士兰科技大学（QUT））
We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies. We apply our benchmark to two simulation environments, PyBullet and V-Rep, and publish the results.…
Technical University of Darmstadt（达姆施塔特工业大学）；University of Washington（华盛顿大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Cornell University（康奈尔大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Dexterous manipulation is a challenging and important problem in robotics. While data-driven methods are a promising approach, current benchmarks require simulation or extensive engineering support due to the sample inefficiency of popular methods. We present benchmarks for the TriFinger system, an open-source robotic platform for dexterous manipulation and the focus of the 2020 Real Robot Challenge. The benchmarked methods, which were successful in the challenge, can be generally described as structured policies, as they combine elements of classical robotics and modern policy optimization. T…
ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Rice University（里士大学）；U.S. National Institute of Standards and Technology（美国国家标准与技术研究院）；Massachusetts Institute of Technology（麻省理工学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄技术大学）；Autodesk Research（Autodesk研究）；University of California, Berkeley（加州大学伯克利分校）；KTH Royal Institute of Technology（皇家理工学院）；Tsinghua University（清华大学）；Columbia University（哥伦比亚大学）；ASTM International（美国材料与试验协会）；Carnegie Mellon University（卡内基梅隆大学）；German Aerospace Center (DLR)（德国航空航天中心）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA Research（NVIDIA研究）
Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Shanghai Jiaotong University（上海交通大学）；Nanjing University（南京大学）
DexBioLab: Benchmarking Dexterous Manipulation with Dynamic Experimental Phenomena in Realistic Laboratories
Shanghai AI Laboratory（上海人工智能实验室）；Huazhong University of Science and Technology（华中科技大学）；Shanghai Jiao Tong University（上海交通大学）；University of Southern California（南加州大学）
Benchmarks offer a scientific way to compare algorithms using objective performance metrics. Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings. In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility. If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general. On the other hand, a benchmark could be a loose set of protocols (e.g. object sets) but the underlying variation in setups make t…
Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation
Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；China University of Mining Technology（中国矿业大学）
Benchmarks for robot manipulation are crucial to measuring progress in the field, yet there are few benchmarks that demonstrate critical manipulation skills, possess standardized metrics, and can be attempted by a wide array of robot platforms. To address a lack of such benchmarks, we propose Rubik's cube manipulation as a benchmark to measure simultaneous performance of precise manipulation and sequential manipulation. The sub-structure of the Rubik's cube demands precise positioning of the robot's end effectors, while its highly reconfigurable nature enables tasks that require the robot to m…
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Department of Computing Science, University of Alberta（阿尔伯塔大学计算机科学系）；National Research Council Canada（加拿大国家研究委员会）；School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）；Vector Institute（向量研究所）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）
Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Nanjing University（南京大学）；Central South University（中南大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Normal University（北京师范大学）
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身智能重点实验室）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）；Neote AI；China Unicom（中国联通）
KTH Royal Institute of Technology（瑞典皇家理工学院）；Yale University（耶鲁大学）
This work focuses on the problem of in-hand manipulation and regrasping of objects with parallel grippers. We propose Dexterous Manipulation Graph (DMG) as a representation on which we define planning for in-hand manipulation and regrasping. The DMG is a disconnected undirected graph that represents the possible motions of a finger along the object's surface. We formulate the in-hand manipulation and regrasping problem as a graph search problem from the initial to the final configuration. The resulting plan is a sequence of coordinated in-hand pushing and regrasping movements. We propose a dua…
Benchmarking Dexterity of Multifingered Robot Hands: A Review and Perspective
US National Science Foundation HAND Engineering Research Center（美国国家科学基金会HAND工程研究中心）；Center for Robotics and Biosystems, Northwestern Univ.（西北大学机器人与生物系统中心）；Department of Mechanical Engineering, Texas A&M Univ.（德克萨斯A&M大学机械工程系）；Department of Electrical and Computer Engineering, Carnegie Mellon Univ.（卡内基梅隆大学电子与计算机工程系）；Department of Mechanical Engineering, Florida A&M Univ.（佛罗里达A&M大学机械工程系）
Osaka University（大阪大学）；Meiji University（明治大学）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
Synergy supplies a practical approach for expressing various postures of a multi-fingered hand. However, a conventional synergy defined for reproducing grasping postures cannot perform general-purpose tasks expected for a multi-fingered hand. Locking the position of particular fingers is essential for a multi-fingered hand to manipulate an object. When using conventional synergy based control to manipulate an object, which requires locking some fingers, the coordination of joints is heavily restricted, decreasing the dexterity of the hand. We propose a functionally divided manipulation synergy…
Dexterous Soft Hands Linearize Feedback-Control for In-Hand Manipulation
Robotics and Biology Laboratory, Technische Universität Berlin（柏林工业大学机器人与生物实验室）；Science of Intelligence, Research Cluster of Excellence, Berlin（柏林卓越研究集群智能科学）
Cross-Embodiment Robot Manipulation via a Unified Hand Action Space
Intelligent Robotics and Vision Lab, University of Texas at Dallas（德克萨斯大学达拉斯分校智能机器人与视觉实验室）；Intelligent Robotics and Interactive Systems Lab, Arizona State University（亚利桑那州立大学智能机器人与交互系统实验室）
Worcester Polytechnic Institute（伍斯特理工学院）；Imperial College London（伦敦帝国理工学院）
Attempts to achieve robotic Within-Hand-Manipulation (WIHM) generally utilize either high-DOF robotic hands with elaborate sensing apparatus or multi-arm robotic systems. In prior work we presented a simple robot hand with variable friction robot fingers, which allow a low-complexity approach to within-hand object translation and rotation, though this manipulation was limited to planar actions. In this work we extend the capabilities of this system to 3D manipulation with a novel region-based WIHM planning algorithm and utilizing extrinsic contacts. The ability to modulate finger friction enha…
Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize…
We present in-hand manipulation skills on a dexterous, compliant, anthropomorphic hand. Even though these skills were derived in a simplistic manner, they exhibit surprising robustness to variations in shape, size, weight, and placement of the manipulated object. They are also very insensitive to variation of execution speeds, ranging from highly dynamic to quasi-static. The robustness of the skills leads to compositional properties that enable extended and robust manipulation programs. To explain the surprising robustness of the in-hand manipulation skills, we performed a detailed, empirical …
Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（德国航空航天中心（DLR）机器人与机电一体化研究所）；Deggendorf Institute of Technology（代根多夫应用技术大学）
We show that a purely tactile dextrous in-hand manipulation task with continuous regrasping, requiring permanent force closure, can be learned from scratch and executed robustly on a torque-controlled humanoid robotic hand. The task is rotating a cube without dropping it, but in contrast to OpenAI's seminal cube manipulation task, the palm faces downwards and no cameras but only the hand's position and torque sensing are used. Although the task seems simple, it combines for the first time all the challenges in execution as well as learning that are important for using in-hand manipulation in r…
Progressive Transfer Learning for Dexterous In-Hand Manipulation with Multi-Fingered Anthropomorphic Hand
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）
TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot
Southern University of Science and Technology（南方科技大学）；University of Hong Kong（香港大学）；Tencent Robotics X（腾讯 Robotics X）；University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）
