Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents
ETH Zürich（苏黎世联邦理工学院）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；Mila, Université de Montréal（蒙特利尔大学米拉研究所）
Jacopo Tani, Andrea F. Daniele, Gianmarco Bernasconi, Amaury Camus, Aleksandar Petrov, Anthony Courchesne, Bhairav Mehta, Rohit Suri, Tomasz Zaluska, Matthew R.… 展开作者
Jacopo Tani, Andrea F. Daniele, Gianmarco Bernasconi, Amaury Camus, Aleksandar Petrov, Anthony Courchesne, Bhairav Mehta, Rohit Suri, Tomasz Zaluska, Matthew R. Walter, Emilio Frazzoli, Liam Paull, Andrea Censi
As robotics matures and increases in complexity, it is more necessary than ever that robot autonomy research be reproducible. Compared to other sciences, there are specific challenges to benchmarking autonomy, such as the complexity of the software stacks, the variability of the hardware and the reliance on data-driven techniques, amongst others. In this paper, we describe a new concept for reproducible robotics research that integrates development and benchmarking, so that reproducibility is obtained "by design" from the beginning of the research/development processes. We first provide the overall conceptual objectives to achieve this goal and then a concrete instance that we have built: the DUCKIENet. One of the central components of this setup is the Duckietown Autolab, a remotely accessible standardized setup that is itself also relatively low-cost and reproducible. When evaluating agents, careful definition of interfaces allows users to choose among local versus remote evaluation using simulation, logs, or remote automated hardware setups. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.
A reproducible and extensible framework for benchmarking competing risks survival models
Cancer Research UK Scotland Centre（英国癌症研究苏格兰中心）；Institute of Genetics and Cancer（遗传与癌症研究所）；University of Edinburgh（爱丁堡大学）；University of Augsburg（奥格斯堡大学）
Reproducible Benchmarking for Lung Nodule Detection and Malignancy Classification Across Multiple Low-Dose CT Datasets
Duke University School of Medicine（杜克大学医学院）；Pratt School of Engineering, Duke University（杜克大学普拉特工程学院）；University of Colorado（科罗拉多大学）；Duke University（杜克大学）
School of Technology Management and Engineering, NMIMS（NMIMS技术管理与工程学院）；Indian Institute of Technology, Roorkee（鲁尔基印度理工学院）；Birla Institute of Technology and Science, Pilani (BITS Pilani)（皮拉尼贝拉理工学院（BITS Pilani））
Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers
Institute for Biomedical Informatics, University of Pennsylvania（宾夕法尼亚大学生物医学信息学研究所）；AGH University of Science and Technology（AGH科技大学）
了解机器学习（ML）算法的优点和缺点对于确定其应用范围至关重要。在这里，我们介绍了DIverse和GENerative ML Benchmark（DIGEN）-一个综合数据集的集合，用于对二进制结果分类的机器学习算法进行全面，可重复和可解释的基准测试。DIGEN资源由40个数学函数组成，这些函数将连续特征映射到离散端点，以创建合成数据集。这40个函数是使用启发式算法发现的，该算法旨在最大限度地提高多个流行的机器学习算法之间的性能差异，从而为评估和比较新方法提供了有用的测试套件。访问生成函数有助于理解为什么一种方法与其他算法相比表现不佳，从而提供改进的想法。包含大量文档和分析的资源是开源的，可在GitHub上获得。
LMU Munich（慕尼黑大学）；University of Freiburg（弗莱堡大学）；Eindhoven University of Technology（埃因霍温理工大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）；TU Dortmund University（多特蒙德工业大学）；Federal Technology University Paraná (UTFPR)（巴拉那联邦科技大学）；Leiden University（莱顿大学）
SynthCharge: An Electric Vehicle Routing Instance Generator with Feasibility Screening to Enable Learning-Based Optimization and Benchmarking
Industrial and Systems Engineering, University of Miami, Coral Gables, FL, USA（迈阿密大学工业与系统工程系）；Faculty of Engineering, Cyprus International University 99258 Nicosia, North Cyprus, via Mersin 10, Turkey（塞浦路斯国际大学工程学院）；Engineering Management and Systems Engineering, Missouri University of Science and Technology, Rolla, MO, USA（密苏里科学与技术大学工程管理与系统工程系）
SMapper: A Multi-Modal Data Acquisition Platform for SLAM Benchmarking
Automation and Robotics Research Group (ARG), Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg（卢森堡大学自动化与机器人研究组（ARG）、安全、可靠与信任跨学科研究中心（SnT））
A ROS2 Benchmarking Framework for Hierarchical Control Strategies in Mobile Robots for Mediterranean Greenhouses
Department of Informatics, CIESOL, ceiA3, Universidad de Almería（信息学院、CIESOL、ceiA3、阿尔梅里亚大学）；Department of Computer Science and Automation, National University of Distance Education (UNED)（计算机科学与自动化系、远程教育国家大学（UNED））；Department of Engineering, CIESOL, ceiA3, Universidad de Almería（工程学院、CIESOL、ceiA3、阿尔梅里亚大学）
