arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2206.05985cs.LGstat.ME
收藏

机器学习多元宇宙建模

Modeling the Machine Learning Multiverse

University of Cambridge(剑桥大学);Allen Institute for AI(艾伦人工智能研究所)

Samuel J. Bell, Onno P. Kampman, Jesse Dodge, Neil D. Lawrence

摘要:

在对机器学习研究的可靠性和可信度越来越多的关注中,我们提出了一个原则性的框架,用于提出强大的和可推广的主张:多元宇宙分析。我们的框架建立在多元宇宙分析的基础上(Steegen et al.,2016年),在回应心理学自身的再现性危机。为了有效地探索高维且通常连续的ML搜索空间,我们使用高斯过程代理对多元宇宙进行建模,并应用贝叶斯实验设计。我们的框架旨在促进对模型性能得出可靠的科学结论,因此我们的方法侧重于探索而不是传统的优化。在两个案例研究的第一个,我们调查有争议的索赔自适应优化的相对优点。其次,我们综合了学习率对大批量训练泛化差距影响的相互矛盾的研究。对于机器学习社区来说,多元宇宙分析是一种简单而有效的技术,用于识别可靠的声明,提高透明度,并朝着提高可重复性迈出了一步。

英文摘要:

Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis. Our framework builds upon the multiverse analysis (Steegen et al., 2016) introduced in response to psychology's own reproducibility crisis. To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design. Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization. In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers. Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap. For the machine learning community, the multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑