arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1912.09536cs.LGcs.DCstat.ML
收藏

数据科学通过镜子和我们在那里发现了什么

Data Science through the looking glass and what we found there

Microsoft(微软)

Fotis Psallidas, Yiwen Zhu, Bojan Karlas, Matteo Interlandi, Avrilia Floratou, Konstantinos Karanasos, Wentao Wu, Ce Zhang, Subru Krishnan, Carlo Curino, Markus Weimer

摘要:

机器学习(ML)最近的成功导致了工业和学术内置的的新系统和算法以及不断增长的数据科学(DS)从业者社区构建的新应用程序的爆炸性增长。这种快速变化的技术和应用全景对于构建者和从业者来说都是一个挑战。在本文中,我们开始通过广角镜头捕捉这一全景,通过对DS项目进行迄今为止最大的分析,重点关注可以帮助确定双方投资的问题。具体来说,我们下载并分析了:(a)GSPOT UB上公开的超过600万个Python笔记本,(b)COMPANYX内开发的超过200万个企业DS管道,以及(c)来自12个重要DS库的900多个版本的源代码和元数据。我们执行的分析范围从粗粒度的统计特征到库导入分析、管道分析以及跨数据集和时间的比较研究。我们报告了大量的测量数据供读者解读,并大胆地得出一些(可操作的,但主观的)结论:(a)系统构建者应该关注什么,以更好地为从业者服务,以及(b)鉴于当前的趋势,从业者应该押注什么技术。我们计划自动化此分析,并定期发布相关工具和结果。

英文摘要:

The recent success of machine learning (ML) has led to an explosive growth both in terms of new systems and algorithms built in industry and academia, and new applications built by an ever-growing community of data science (DS) practitioners. This quickly shifting panorama of technologies and applications is challenging for builders and practitioners alike to follow. In this paper, we set out to capture this panorama through a wide-angle lens, by performing the largest analysis of DS projects to date, focusing on questions that can help determine investments on either side. Specifically, we download and analyze: (a) over 6M Python notebooks publicly available on GITHUB, (b) over 2M enterprise DS pipelines developed within COMPANYX, and (c) the source code and metadata of over 900 releases from 12 important DS libraries. The analysis we perform ranges from coarse-grained statistical characterizations to analysis of library imports, pipelines, and comparative studies across datasets and time. We report a large number of measurements for our readers to interpret, and dare to draw a few (actionable, yet subjective) conclusions on (a) what systems builders should focus on to better serve practitioners, and (b) what technologies should practitioners bet on given current trends. We plan to automate this analysis and release associated tools and results periodically.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑