Benchmarks offer a scientific way to compare algorithms using objective performance metrics. Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings. In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility. If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general. On the other hand, a benchmark could be a loose set of protocols (e.g. object sets) but the underlying variation in setups make the results non-reproducible. In this paper, we re-imagine benchmarking for robotic manipulation as state-of-the-art algorithmic implementations, alongside the usual set of tasks and experimental protocols. The added baseline implementations will provide a way to easily recreate SOTA numbers in a new local robotic setup, thus providing credible relative rankings between existing approaches and new work. However, these local rankings could vary between different setups. To resolve this issue, we build a mechanism for pooling experimental data between labs, and thus we establish a single global ranking for existing (and proposed) SOTA algorithms. Our benchmark, called Ranking-Based Robotics Benchmark (RB2), is evaluated on tasks that are inspired from clinically validated Southampton Hand Assessment Procedures. Our benchmark was run across two different labs and reveals several surprising findings. For example, extremely simple baselines like open-loop behavior cloning, outperform more complicated models (e.g. closed loop, RNN, Offline-RL, etc.) that are preferred by the field. We hope our fellow researchers will use RB2 to improve their research's quality and rigor.
KTH Royal Institute of Technology（KTH皇家理工学院）；University of Utah（犹他大学）；Yale University（耶鲁大学）；Google AI（谷歌人工智能）；NVIDIA Research（英伟达研究院）
The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states (i.e.\ static object poses and fingertip locations) for various in-hand manipulation tasks. We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across a…
Developing Modular Grasping and Manipulation Pipeline Infrastructure to Streamline Performance Benchmarking
New England Robotics Validation and Experimentation (NERVE) Center, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；Rutgers University（罗格斯大学）；Worcester Polytechnic University（伍斯特理工大学）；Yale University（耶鲁大学）；University of South Florida（南佛罗里达大学）
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息学研究所）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Benchmarking of robotic manipulations is one of the open issues in robotic research. An important factor that has enabled progress in this area in the last decade is the existence of common object sets that have been shared among different research groups. However, the existing object sets are very limited when it comes to cloth-like objects that have unique particularities and challenges. This paper is a first step towards the design of a cloth object set to be distributed among research groups from the robotics cloth manipulation community. We present a set of household cloth objects and rel…
Data61, Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO）Data61研究所）；Queensland University of Technology (QUT)（昆士兰科技大学（QUT））
We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies. We apply our benchmark to two simulation environments, PyBullet and V-Rep, and publish the results.…
Technical University of Darmstadt（达姆施塔特工业大学）；University of Washington（华盛顿大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Cornell University（康奈尔大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Dexterous manipulation is a challenging and important problem in robotics. While data-driven methods are a promising approach, current benchmarks require simulation or extensive engineering support due to the sample inefficiency of popular methods. We present benchmarks for the TriFinger system, an open-source robotic platform for dexterous manipulation and the focus of the 2020 Real Robot Challenge. The benchmarked methods, which were successful in the challenge, can be generally described as structured policies, as they combine elements of classical robotics and modern policy optimization. T…
ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Rice University（里士大学）；U.S. National Institute of Standards and Technology（美国国家标准与技术研究院）；Massachusetts Institute of Technology（麻省理工学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄技术大学）；Autodesk Research（Autodesk研究）；University of California, Berkeley（加州大学伯克利分校）；KTH Royal Institute of Technology（皇家理工学院）；Tsinghua University（清华大学）；Columbia University（哥伦比亚大学）；ASTM International（美国材料与试验协会）；Carnegie Mellon University（卡内基梅隆大学）；German Aerospace Center (DLR)（德国航空航天中心）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA Research（NVIDIA研究）
Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Shanghai Jiaotong University（上海交通大学）；Nanjing University（南京大学）
DexBioLab: Benchmarking Dexterous Manipulation with Dynamic Experimental Phenomena in Realistic Laboratories
Shanghai AI Laboratory（上海人工智能实验室）；Huazhong University of Science and Technology（华中科技大学）；Shanghai Jiao Tong University（上海交通大学）；University of Southern California（南加州大学）
Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation
Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；China University of Mining Technology（中国矿业大学）
Benchmarks for robot manipulation are crucial to measuring progress in the field, yet there are few benchmarks that demonstrate critical manipulation skills, possess standardized metrics, and can be attempted by a wide array of robot platforms. To address a lack of such benchmarks, we propose Rubik's cube manipulation as a benchmark to measure simultaneous performance of precise manipulation and sequential manipulation. The sub-structure of the Rubik's cube demands precise positioning of the robot's end effectors, while its highly reconfigurable nature enables tasks that require the robot to m…
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Department of Computing Science, University of Alberta（阿尔伯塔大学计算机科学系）；National Research Council Canada（加拿大国家研究委员会）；School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）；Vector Institute（向量研究所）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）
Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Nanjing University（南京大学）；Central South University（中南大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Normal University（北京师范大学）
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身智能重点实验室）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）；Neote AI；China Unicom（中国联通）
Benchmarking Dexterity of Multifingered Robot Hands: A Review and Perspective
US National Science Foundation HAND Engineering Research Center（美国国家科学基金会HAND工程研究中心）；Center for Robotics and Biosystems, Northwestern Univ.（西北大学机器人与生物系统中心）；Department of Mechanical Engineering, Texas A&M Univ.（德克萨斯A&M大学机械工程系）；Department of Electrical and Computer Engineering, Carnegie Mellon Univ.（卡内基梅隆大学电子与计算机工程系）；Department of Mechanical Engineering, Florida A&M Univ.（佛罗里达A&M大学机械工程系）
Identification of textile properties is an important milestone toward advanced robotic manipulation tasks that consider interaction with clothing items such as assisted dressing, laundry folding, automated sewing, textile recycling and reusing. Despite the abundance of work considering this class of deformable objects, many open problems remain. These relate to the choice and modelling of the sensory feedback as well as the control and planning of the interaction and manipulation strategies. Most importantly, there is no structured approach for studying and assessing different approaches that …
RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation
School of Cyber Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；National University of Singapore（新加坡国立大学）；Sun Yat-sen University（中山大学）；Beihang University（北京航空航天大学）
Towards Accurate End-Effector Localization for UMI-Style Robotic Manipulation Teaching
Chongqing University（重庆大学）；Northwestern Polytechnical University（西北工业大学）；The University of Tokyo（东京大学）；Jiaxing University（嘉兴大学）；Sichuan University（四川大学）；Beijing Institute of Technology（北京理工大学）；Shanghai Jiao Tong University（上海交通大学）
Worcester Polytechnic Institute（伍斯特理工学院）；Imperial College London（伦敦帝国理工学院）
Attempts to achieve robotic Within-Hand-Manipulation (WIHM) generally utilize either high-DOF robotic hands with elaborate sensing apparatus or multi-arm robotic systems. In prior work we presented a simple robot hand with variable friction robot fingers, which allow a low-complexity approach to within-hand object translation and rotation, though this manipulation was limited to planar actions. In this work we extend the capabilities of this system to 3D manipulation with a novel region-based WIHM planning algorithm and utilizing extrinsic contacts. The ability to modulate finger friction enha…
University of California, Riverside（加州大学河滨分校）；Yale University（耶鲁大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；University of Toronto（多伦多大学）
DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation
Autonomous Systems Lab, Institute of Computer Technology, TU Wien（自主系统实验室，计算机技术学院，维也纳技术大学）；Institute of Robotics and Mechatronics (DLR), German Aerospace Center（机器人与机电系统研究所（DLR），德国航空航天中心）
A Benchmarking Study of Vision-Based Robotic Grasping Algorithms: A Comparative Analysis
Robotics Engineering Department, Worcester Polytechnic Institute（沃斯特理工学院机器人工程系）；New England Robotics Validation and Experimentation (NERVE) Center, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校新英格兰机器人验证与实验中心）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Robotic systems that can dynamically combine manipulation and locomotion could facilitate dangerous or physically demanding labor. For instance, firefighter humanoid robots could leverage their body by leaning against collapsed building rubble to push it aside. Here we introduce a teleoperation system that targets the realization of these tasks using human whole-body motor skills. We describe a new wheeled humanoid platform, SATYRR, and a novel hands-free teleoperation architecture using a whole-body Human Machine Interface (HMI). This system enables telelocomotion of the humanoid robot using …
Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success
Faculty of Electrical Engineering, Czech Technical University in Prague（捷克技术大学布拉格电子工程系）；Czech Institute of Informatics, Robotics and Cybernetics（捷克信息学、机器人学与自动化研究所）
Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation
South China University of Technology（华南理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Northwestern Polytechnical University（西北工业大学）；The Hong Kong Polytechnic University（香港理工大学）
Imperial College London（伦敦帝国理工学院）；University of Pennsylvania（宾夕法尼亚大学）
Robotic dual-arm twisting is a common but very challenging task in both industrial production and daily services, as it often requires dexterous collaboration, a large scale of end-effector rotating, and good adaptivity for object manipulation. Meanwhile, safety and efficiency are preliminary concerns for robotic dual-arm coordinated manipulation. Thus, the normally adopted fully automated task execution approaches based on environmental perception and motion planning techniques are still inadequate and problematic for the arduous twisting tasks. To this end, this paper presents a novel strate…
National University of Singapore（新加坡国立大学）；Sea AI Lab（Sea人工智能实验室）
Deformable Object Manipulation (DOM) is of significant importance to both daily and industrial applications. Recent successes in differentiable physics simulators allow learning algorithms to train a policy with analytic gradients through environment dynamics, which significantly facilitates the development of DOM algorithms. However, existing DOM benchmarks are either single-object-based or non-differentiable. This leaves the questions of 1) how a task-specific algorithm performs on other tasks and 2) how a differentiable-physics-based algorithm compares with the non-differentiable ones in ge…
