arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.03098cs.LGstat.ML
收藏

解释机器学习基准中的差异

Accounting for Variance in Machine Learning Benchmarks

Mila(米拉(学习算法研究所));Université de Montréal(蒙特利尔大学);IRIC(免疫和癌症研究所);McGill University(麦吉尔大学);Booz Allen Hamilton(博思艾伦汉密尔顿公司);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);École de technologie supérieure(高等技术学院);CIFAR(加拿大高级研究所);Polytechnique Montréal(蒙特利尔理工学院);ElementAI(ElementAI公司);Inria(法国国家信息与自动化研究所)
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira… 展开作者

Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, Pascal Vincent

摘要:

强有力的经验证据表明,一种机器学习算法A优于另一种算法B,理想情况下需要多次试验来优化学习管道的变化来源,如数据采样,数据增强,参数初始化和超参数选择。这是令人望而却步的昂贵,并抄近路得出结论。我们对整个基准测试过程进行了建模,揭示了数据采样、参数初始化和超参数选择引起的方差对结果有显着影响。我们根据这种差异分析了当今使用的主要比较方法。我们展示了一个反直觉的结果,即向不完美估计量添加更多的变异源可以更好地接近理想估计量,计算成本降低51倍。在这些结果的基础上,我们研究了在五种不同的深度学习任务/架构上检测改进的出错率。这项研究使我们提出了性能比较的建议。

英文摘要:

Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑