We present DeepClaw as a reconfigurable benchmark of robotic hardware and task hierarchy for robot learning. The DeepClaw benchmark aims at a mechatronics perspective of the robot learning problem, which features a minimum design of robot cell that can be easily reconfigured to host robot hardware from various vendors, including manipulators, grippers, cameras, desks, and objects, aiming at a streamlined collection of physical manipulation data and evaluation of the learned skills for hardware benchmarking. We provide a detailed design of the robot cell with readily available parts to build the experiment environment that can host a wide range of robotic hardware commonly adopted for robot learning. We also propose a hierarchical pipeline of software integration, including localization, recognition, grasp planning, and motion planning, to streamline learning-based robot control, data collection, and experiment validation towards shareability and reproducibility. We present benchmarking results of the DeepClaw system for a baseline Tic-Tac-Toe task, a bin-clearing task, and a jigsaw puzzle task using three sets of standard robotic hardware. Our results show that tasks defined in DeepClaw can be easily reproduced on three robot cells. Under the same task setup, the differences in robotic hardware used will present a non-negligible impact on the performance metrics of robot learning. All design layouts and codes are hosted on Github for open access.
RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control
Tsinghua University（清华大学）；University at Buffalo, State University of New York（纽约州立大学布法罗分校）；Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research Asia（微软亚洲研究院）；Chengdu University of Information Technology（成都信息工程大学）
Developing Modular Grasping and Manipulation Pipeline Infrastructure to Streamline Performance Benchmarking
New England Robotics Validation and Experimentation (NERVE) Center, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；Rutgers University（罗格斯大学）；Worcester Polytechnic University（伍斯特理工大学）；Yale University（耶鲁大学）；University of South Florida（南佛罗里达大学）
Household Cloth Object Set: Fostering Benchmarking in Deformable Object Manipulation
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息学研究所）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Benchmarking Simulated Robotic Manipulation through a Real World Dataset
Data61, Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO）Data61研究所）；Queensland University of Technology (QUT)（昆士兰科技大学（QUT））
Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation
Technical University of Darmstadt（达姆施塔特工业大学）；University of Washington（华盛顿大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Cornell University（康奈尔大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
Rice University（里士大学）；U.S. National Institute of Standards and Technology（美国国家标准与技术研究院）；Massachusetts Institute of Technology（麻省理工学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄技术大学）；Autodesk Research（Autodesk研究）；University of California, Berkeley（加州大学伯克利分校）；KTH Royal Institute of Technology（皇家理工学院）；Tsinghua University（清华大学）；Columbia University（哥伦比亚大学）；ASTM International（美国材料与试验协会）；Carnegie Mellon University（卡内基梅隆大学）；German Aerospace Center (DLR)（德国航空航天中心）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA Research（NVIDIA研究）
Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Shanghai Jiaotong University（上海交通大学）；Nanjing University（南京大学）
DexBioLab: Benchmarking Dexterous Manipulation with Dynamic Experimental Phenomena in Realistic Laboratories
Shanghai AI Laboratory（上海人工智能实验室）；Huazhong University of Science and Technology（华中科技大学）；Shanghai Jiao Tong University（上海交通大学）；University of Southern California（南加州大学）
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
ScaleLab, Shanghai Jiao Tong University（上海交通大学ScaleLab）；D-Robotics；ViTai Robotics；The University of Hong Kong（香港大学）；Nanjing University（南京大学）；Shenzhen University（深圳大学）；Wuhan University（武汉大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation
Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；China University of Mining Technology（中国矿业大学）
Bench-Push: Benchmarking Pushing-based Navigation and Manipulation Tasks for Mobile Robots
Institute for Robotics and Intelligent Machines, Georgia Institute of Technology（机器人与智能机器研究所，佐治亚理工学院）；Department of Electrical and Computer Engineering, University of Waterloo（电气与计算机工程系，滑铁卢大学）；Department of Mechanical Engineering, University of Alberta（机械工程系，阿尔伯塔大学）
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Department of Computing Science, University of Alberta（阿尔伯塔大学计算机科学系）；National Research Council Canada（加拿大国家研究委员会）；School of Electrical Engineering and Computer Science, University of Ottawa（渥太华大学电气工程与计算机科学学院）；Vector Institute（向量研究所）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）
Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Nanjing University（南京大学）；Central South University（中南大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Normal University（北京师范大学）
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身智能重点实验室）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）；Neote AI；China Unicom（中国联通）
Benchmarking Dexterity of Multifingered Robot Hands: A Review and Perspective
US National Science Foundation HAND Engineering Research Center（美国国家科学基金会HAND工程研究中心）；Center for Robotics and Biosystems, Northwestern Univ.（西北大学机器人与生物系统中心）；Department of Mechanical Engineering, Texas A&M Univ.（德克萨斯A&M大学机械工程系）；Department of Electrical and Computer Engineering, Carnegie Mellon Univ.（卡内基梅隆大学电子与计算机工程系）；Department of Mechanical Engineering, Florida A&M Univ.（佛罗里达A&M大学机械工程系）
