arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1904.12437cs.LGcs.AIcs.SE
收藏

机器学习评估和基准测试的挑战和陷阱

Challenges and Pitfalls of Machine Learning Evaluation and Benchmarking

University of Illinois, Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);IBM Thomas J. Watson Research Center(IBM托马斯·J·沃森研究中心)

Cheng Li, Abdul Dakkak, Jinjun Xiong, Wen-mei Hwu

摘要:

越来越复杂和多样化的机器学习(ML)模型以及硬件/软件堆栈(统称为“ML工件”)正在被提出-导致ML的多样化景观。提出的这些ML创新已经超过了研究人员分析,研究和适应它们的能力。这是加剧了复杂的,有时不可重现的ML评估程序。共享ML工件的一个常见做法是通过存储库,工件作者在存储库中发布临时代码和一些文档,但通常无法为其他人提供关键信息以重现其结果。这导致用户无法与工件作者的声明进行比较,或者使模型适应他/她自己的使用。本文讨论了ML评估和基准测试的常见挑战和陷阱,可用作ML模型作者共享ML工件时的指南,以及系统开发人员在基准测试或设计ML系统时的指南。

英文摘要:

An increasingly complex and diverse collection of Machine Learning (ML) models as well as hardware/software stacks, collectively referred to as "ML artifacts", are being proposed - leading to a diverse landscape of ML. These ML innovations proposed have outpaced researchers' ability to analyze, study and adapt them. This is exacerbated by the complicated and sometimes non-reproducible procedures for ML evaluation. A common practice of sharing ML artifacts is through repositories where artifact authors post ad-hoc code and some documentation, but often fail to reveal critical information for others to reproduce their results. This results in users' inability to compare with artifact authors' claims or adapt the model to his/her own use. This paper discusses common challenges and pitfalls of ML evaluation and benchmarking, which can be used as a guideline for ML model authors when sharing ML artifacts, and for system developers when benchmarking or designing ML systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑