An increasingly complex and diverse collection of Machine Learning (ML) models as well as hardware/software stacks, collectively referred to as "ML artifacts", are being proposed - leading to a diverse landscape of ML. These ML innovations proposed have outpaced researchers' ability to analyze, study and adapt them. This is exacerbated by the complicated and sometimes non-reproducible procedures for ML evaluation. A common practice of sharing ML artifacts is through repositories where artifact authors post ad-hoc code and some documentation, but often fail to reveal critical information for others to reproduce their results. This results in users' inability to compare with artifact authors' claims or adapt the model to his/her own use. This paper discusses common challenges and pitfalls of ML evaluation and benchmarking, which can be used as a guideline for ML model authors when sharing ML artifacts, and for system developers when benchmarking or designing ML systems.
Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking
Rensselaer Polytechnic Institute(罗切斯特理工学院);University of Toronto(多伦多大学);Pennsylvania State University(宾夕法尼亚州立大学);AT&T Chief Data Office(AT&T首席数据办公室);Griffith University(格里菲斯大学)
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Microsoft(微软);University College London(伦敦大学学院);University of Illinois, UC(伊利诺伊大学厄巴纳-香槟分校);University of Waterloo(滑铁卢大学)
TREC,CLEF,NTCIR和FIRE等评估工作,以及MS MARCO等公共排行榜,旨在鼓励研究和跟踪我们的进展,解决我们领域的重大问题。然而,目标不仅仅是确定哪个运行是“最好的”,获得最高分数。我们的目标是通过开发新的强大的技术来推动该领域的发展,这些技术在许多不同的环境中工作,并在研究和实践中采用。本文使用MS MARCO和TREC深度学习Track作为我们的案例研究,将其与20世纪90年代的TREC ad hoc排名案例进行比较。我们展示了如何设计的评价工作可以鼓励或阻止某些结果,并提出了内部和外部的有效性的结果的问题。我们对某些陷阱进行了一些分析,并陈述了避免此类陷阱的最佳实践。我们总结了到目前为止的工作进展,并描述了我们期望的“强大的有用性”的最终状态,以及可能需要采取的步骤。
Benchmarking Machine Learning Models to Predict Corporate Bankruptcy
McCoy College of Business at Texas State University(德克萨斯州立大学麦考伊商学院);Scheller College of Business at Georgia Institute of Technology(佐治亚理工学院谢勒商学院);College of Computing at Georgia Institute of Technology(佐治亚理工学院计算学院)
PMLB v1.0: An open source dataset collection for benchmarking machine learning methods
University of Pennsylvania(宾夕法尼亚大学);Washington University in St. Louis(圣路易斯华盛顿大学);Princeton Day School(普林斯顿走读学校)
动机:新的机器学习和统计建模研究依赖于使用经过充分研究的基准数据集与现有方法进行标准化比较。很少有工具可以通过标准化的、用户友好的界面快速访问这些数据集,该界面与流行的数据科学工作流集成良好。结果如下:此版本的PMLB提供了最大的多样化公共基准数据集集合,用于评估聚合在一个位置的新机器学习和数据科学方法。v1.0引入了在与开放源代码社区讨论后开发的一些关键改进。可用性:PMLB可在https://github.com/EpistasisLab/pmlb上获得。PMLB的Python和R接口可以分别通过Python包索引和Comprehensive R Archive Network安装。
On Benchmarking Human-Like Intelligence in Machines
Harvard University(哈佛大学);Massachusetts Institute of Technology(麻省理工学院);University of Cambridge(剑桥大学);New York University(纽约大学);Princeton University(普林斯顿大学);Johns Hopkins University(约翰斯·霍普金斯大学);Stanford University(斯坦福大学)
Machine-learning for photoplethysmography analysis: Benchmarking feature, image, and signal-based approaches
Carl von Ossietzky Universität Oldenburg(奥尔登堡卡尔·冯·奥西特齐大学);Laboratoire national de métrologie et d’essais(国家计量与测试实验室);Biomedical Engineering Institute, Kaunas University of Technology(克莱萨斯理工大学生物医学工程研究所);Physikalisch-Technische Bundesanstalt(德国物理技术部);National Physical Laboratory(国家物理实验室);King’s College London(伦敦国王学院);University of Surrey(萨里大学);Department of Public Health and Primary Care, University of Cambridge(剑桥大学公共卫生与初级保健系)
A Machine Learning Benchmarking Framework for Lipid Nanoparticle Transfection Efficiency Prediction
Department of Electrical and Computer Engineering, University of Southern California(电气与计算机工程系,南加州大学);University of Cambridge(剑桥大学);School of Pharmacy, University of Wisconsin-Madison(威斯康星大学麦迪逊分校药学院);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);East Tennessee State University(东田纳西州立大学)
Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection
Jiangsu Key Laboratory of Big Data Security and Intelligent Processing, Nanjing University of Posts and Telecommunications(江苏大数据安全与智能处理重点实验室,南京邮电大学);University of Brighton(布莱顿大学);University of Lancashire(兰开夏大学);School of Cyber Science and Engineering, Southeast University(东南大学信息科学与工程学院)
Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls
University of Calgary(卡尔加里大学);McGill University Health Centre(麦吉尔大学健康中心);McGill University(麦吉尔大学);Massachusetts General Hospital(麻省总医院);Montreal Imaging Experts(蒙特利尔影像专家公司);Jewish General Hospital(犹太综合医院);University of Florida College of Medicine(佛罗里达大学医学院)
Dos and Don'ts of Machine Learning in Computer Security
Technische Universität Berlin(柏林工业大学);Technische Universität Braunschweig(布伦瑞克工业大学);King’s College London(伦敦国王学院);University College London(伦敦大学学院);Royal Holloway, University of London(伦敦大学皇家霍洛威学院);The Alan Turing Institute(艾伦图灵研究所);KASTEL Security Research Labs(KASTEL安全研究实验室);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院)
MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery
Massachusetts Institute of Technology(麻省理工学院);Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague(捷克信息学、机器人学与自动化捷克技术大学);Enveda Biosciences(Enveda 生物科技);Tufts University(塔夫茨大学)
Developing Machine-Learned Potentials for Coarse-Grained Molecular Simulations: Challenges and Pitfalls
Institute of Informatics and Telecommunications(信息学与电信研究所);Institute of Nanoscience and Nanotechnology(纳米科学与纳米技术研究所);National Centre for Scientific Research “Demokritos”(“德谟克利特”国家科学研究中心);SciFY P.N.P.C.(SciFY公共非营利公司);School of Chemical Engineering(化学工程学院);National Technical University of Athens(雅典国立技术大学)
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
University of Cambridge(剑桥大学);AstraZeneca(阿斯利康);University of Manchester(曼彻斯特大学);Royal Papworth Hospital(皇家帕普沃斯医院);Cambridge University Hospitals NHS Trust(剑桥大学医院国民医疗服务信托)
Beyond algorithm hyperparameters: on preprocessing hyperparameters and associated pitfalls in machine learning applications
Institute for Medical Information Processing, Biometry and Epidemiology, Faculty of Medicine, LMU Munich(医学信息处理、生物统计与流行病学研究所,医学学院,慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Department of Palliative Medicine, University Hospital, LMU Munich(姑息医学部,大学医院,慕尼黑大学);Institute of Clinical Biometrics, Center for Medical Data Science, Medical University of Vienna(临床生物统计研究所,医学数据科学中心,维也纳医学大学)