The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different sources, but their organization and adoption as standards have been inconsistent. As such, selecting and curating specific benchmarks remains an unnecessary burden on machine learning practitioners and data scientists. The present study introduces an accessible, curated, and developing public benchmark resource to facilitate identification of the strengths and weaknesses of different machine learning methodologies. We compare meta-features among the current set of benchmark datasets in this resource to characterize the diversity of available data. Finally, we apply a number of established machine learning methods to the entire benchmark suite and analyze how datasets and algorithms cluster in terms of performance. This work is an important first step towards understanding the limitations of popular benchmarking suites and developing a resource that connects existing benchmarking standards to more diverse and efficient standards in the future.
PMLB v1.0: An open source dataset collection for benchmarking machine learning methods
University of Pennsylvania（宾夕法尼亚大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Princeton Day School（普林斯顿走读学校）
动机：新的机器学习和统计建模研究依赖于使用经过充分研究的基准数据集与现有方法进行标准化比较。很少有工具可以通过标准化的、用户友好的界面快速访问这些数据集，该界面与流行的数据科学工作流集成良好。结果如下：此版本的PMLB提供了最大的多样化公共基准数据集集合，用于评估聚合在一个位置的新机器学习和数据科学方法。v1.0引入了在与开放源代码社区讨论后开发的一些关键改进。可用性：PMLB可在https://github.com/EpistasisLab/pmlb上获得。PMLB的Python和R接口可以分别通过Python包索引和Comprehensive R Archive Network安装。
Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games
University of Cambridge（剑桥大学）；NYU Shanghai（纽约大学上海分校）；NYU Center for Data Science（纽约大学数据科学中心）；Earth Species Project（地球物种计划）；NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai（纽约大学上海分校数学科学研究所）
Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym
Leiden University（莱顿大学）；Meta AI Research（Meta AI 研究院）；Technical University of Munich（慕尼黑工业大学）；LIP6, CNRS, Sorbonne Université（索邦大学国家科学研究中心LIP6实验室）；Universitat Pompeu Fabra（庞培法布拉大学）；Université d’Antananarivo（塔那那利佛大学）
