arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.08430cs.LG
收藏

学习中的可复现性

Reproducibility in Learning

University of California San Diego(加州大学圣地亚哥分校);Columbia University(哥伦比亚大学);IAS(普林斯顿高等研究院);University of Toronto(多伦多大学)

Russell Impagliazzo, Rex Lei, Toniann Pitassi, Jessica Sorrell

AI总结:

本文提出可复现学习算法概念,证明统计查询算法可适度增加样本实现可复现,并构造了学习半空间等问题的首个可复现算法,同时给出近乎紧的样本复杂度上下界。

摘要:

我们在学习语境中引入可复现算法的概念。一个可复现的学习算法对其样本变化具有鲁棒性——当在同一底层分布的两个样本上运行时,它以高概率返回完全相同的输出。我们首先解析这一定义,阐明随机性如何在平衡准确性与可复现性中发挥关键作用。我们开创了可复现算法的理论,展示可复现性如何蕴含数据重用和高效可测试性等理想性质。尽管可复现性的要求极为严格,但在统计学和学习的若干基本问题上仍存在高效的可复现算法。首先,我们证明任何统计查询(SQ)算法都可以通过适度增加样本复杂度而变得可复现,并利用这一点构造了用于寻找近似重击项(heavy-hitters)和中位数的可复现算法。借助这些思想,我们给出了第一个通过可复现弱学习器和可复现提升(boosting)算法学习半空间(halfspaces)的可复现算法。最后,我们开创了对可复现算法下界和内在权衡的研究,给出了可复现与不可复现SQ算法之间近乎紧的样本复杂度上界和下界。

英文摘要:

We introduce the notion of a reproducible algorithm in the context of learning. A reproducible learning algorithm is resilient to variations in its samples -- with high probability, it returns the exact same output when run on two samples from the same underlying distribution. We begin by unpacking the definition, clarifying how randomness is instrumental in balancing accuracy and reproducibility. We initiate a theory of reproducible algorithms, showing how reproducibility implies desirable properties such as data reuse and efficient testability. Despite the exceedingly strong demand of reproducibility, there are efficient reproducible algorithms for several fundamental problems in statistics and learning. First, we show that any statistical query algorithm can be made reproducible with a modest increase in sample complexity, and we use this to construct reproducible algorithms for finding approximate heavy-hitters and medians. Using these ideas, we give the first reproducible algorithm for learning halfspaces via a reproducible weak learner and a reproducible boosting algorithm. Finally, we initiate the study of lower bounds and inherent tradeoffs for reproducible algorithms, giving nearly tight sample complexity upper and lower bounds for reproducible versus nonreproducible SQ algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑