arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.13191cs.SEcs.LG
收藏

应对构建复杂数据系统的挑战:一种开发理念

Navigating the challenges in creating complex data systems: a development philosophy

University of Cambridge(剑桥大学);University of Bremen(不来梅大学);Cambridge University Hospitals NHS Trust(剑桥大学医院国民保健服务信托基金);Addenbrooke’s Hospital(阿登布鲁克医院)

Sören Dittmer, Michael Roberts, Julian Gilbey, Ander Biguri, AIX-COVNET Collaboration, Jacobus Preller, James H. F. Rudd, John A. D. Aston, Carola-Bibiane Schönlieb

AI总结:

本文针对当前数据科学系统可复现性危机,分析了其构建难度高的根源,提出应借鉴软件工程方法,采用增量式开发与正确性、效能双反馈循环的开发理念。

摘要:

在这篇观点文章中,我们提出,尽管过去十年间数据科学与机器学习领域的强大工具已实现普及化,但开发一套可靠且有效的数据科学系统(DSS)的代码难度正不断提升。我们识别出诸多根本原因,其中不当的激励机制与软件工程(SE)技能的普遍匮乏,是导致当前DSS可复现性系统性危机的天然诱因。我们分析了软件工程及大型复杂系统构建总体上难度较高的原因。基于这些洞见,我们明确了软件工程如何应对这些难题,以及我们该如何应用并推广软件工程方法,来构建符合用途的DSS。我们倡导两大核心开发理念:一是应增量式逐步构建DSS,而非采用两阶段式的先规划后搭建模式;二是开发过程中需始终引入两类反馈循环:一类用于测试代码的正确性,另一类用于评估代码的效能。

英文摘要:

In this perspective, we argue that despite the democratization of powerful tools for data science and machine learning over the last decade, developing the code for a trustworthy and effective data science system (DSS) is getting harder. Perverse incentives and a lack of widespread software engineering (SE) skills are among many root causes we identify that naturally give rise to the current systemic crisis in reproducibility of DSSs. We analyze why SE and building large complex systems is, in general, hard. Based on these insights, we identify how SE addresses those difficulties and how we can apply and generalize SE methods to construct DSSs that are fit for purpose. We advocate two key development philosophies, namely that one should incrementally grow -- not biphasically plan and build -- DSSs, and one should always employ two types of feedback loops during development: one which tests the code's correctness and another that evaluates the code's efficacy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑