We introduce the notion of a reproducible algorithm in the context of learning. A reproducible learning algorithm is resilient to variations in its samples -- with high probability, it returns the exact same output when run on two samples from the same underlying distribution. We begin by unpacking the definition, clarifying how randomness is instrumental in balancing accuracy and reproducibility. We initiate a theory of reproducible algorithms, showing how reproducibility implies desirable properties such as data reuse and efficient testability. Despite the exceedingly strong demand of reproducibility, there are efficient reproducible algorithms for several fundamental problems in statistics and learning. First, we show that any statistical query algorithm can be made reproducible with a modest increase in sample complexity, and we use this to construct reproducible algorithms for finding approximate heavy-hitters and medians. Using these ideas, we give the first reproducible algorithm for learning halfspaces via a reproducible weak learner and a reproducible boosting algorithm. Finally, we initiate the study of lower bounds and inherent tradeoffs for reproducible algorithms, giving nearly tight sample complexity upper and lower bounds for reproducible versus nonreproducible SQ algorithms.
Utilizing stability criteria in choosing feature selection methods yields reproducible results in microbiome data
University of California San Diego(加州大学圣地亚哥分校);IBM T. J. Watson Research Center(IBM托马斯·J·沃森研究中心);IBM Research UK(IBM英国研究院);IBM Research-Almaden(IBM阿尔马登研究院)
Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform
Drexel University(德雷塞尔大学);Hugging Face(抱抱脸);VISTEC(泰国国家科技发展局国家科学技术研究院(注:VISTEC 常见译名为泰国国家科技发展局国家科学技术研究院,若需更通用译法可保留缩写直译));Sea AI Lab(Sea人工智能实验室);Google(谷歌);Tencent AI Lab(腾讯人工智能实验室)
From Data to Decision: Data-Centric Infrastructure for Reproducible ML in Collaborative eScience
Dept. of Industrial and Systems Engineering(工业与系统工程系);University of Southern California(南加州大学);Information Sciences Institute(信息科学研究所);Dept. of Computer Science(计算机科学系);Dept. of Ophthalmology(眼科学系)
The NCI Imaging Data Commons as a platform for reproducible research in computational pathology
Fraunhofer Institute for Digital Medicine MEVIS(弗劳恩霍夫数字医学研究所MEVIS);Department of Pathology, Massachusetts General Hospital(麻省总医院病理科部门);Harvard Medical School(哈佛医学院);Institute for Systems Biology(系统生物学研究所)
本文探讨 NCI 影像数据共享平台如何通过提供 FAIR 数据集和云 ML 服务集成,促进计算病理学研究的可重复性,实验显示 AUC 差异最高达 0.045。