arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.16546cs.SEcs.AI
收藏

超越准确性:开源深度学习项目中单元测试的实证研究

Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects

Monash University(莫纳什大学);Jilin University(吉林大学);University of Oulu(奥卢大学)

Han Wang, Sijia Yu, Chunyang Chen, Burak Turhan, Xiaodong Zhu

AI总结:

本文通过对 9,129 个 GitHub 开源深度学习项目的实证分析,揭示其单元测试覆盖率低且与项目质量指标相关,并构建了单元测试与故障的映射分类法,以强调单元测试对深度学习项目可靠性的重要性。

摘要:

深度学习(DL)模型发展迅速,重点在于通过测试模型准确性和鲁棒性来实现高性能。然而,当需要像对待其他软件系统一样对待和测试深度学习项目时,尚不清楚这些项目作为软件系统是否得到了充分测试或在功能上是否正确。因此,我们对开源深度学习项目中的单元测试进行了实证研究,分析了来自 GitHub 的 9,129 个项目。我们发现:1)进行了单元测试的深度学习项目与开源项目指标呈正相关,并且其拉取请求(pull requests)的接受率更高;2)在抽样的深度学习项目中,68% 完全没有进行单元测试;3)深度学习模型的层(layer)和工具(utils)拥有最多的单元测试。基于这些发现和先前的研究成果,我们构建了深度学习项目中单元测试与故障之间的映射分类法。我们讨论了这些发现对开发者和研究人员的启示,并强调了在开源深度学习项目中进行单元测试以确保其可靠性和稳定性的必要性。本研究通过提高人们对深度学习项目中单元测试重要性的认识,并鼓励在这一领域开展进一步研究,为这一社区做出了贡献。

英文摘要:

Deep Learning (DL) models have rapidly advanced, focusing on achieving high performance through testing model accuracy and robustness. However, it is unclear whether DL projects, as software systems, are tested thoroughly or functionally correct when there is a need to treat and test them like other software systems. Therefore, we empirically study the unit tests in open-source DL projects, analyzing 9,129 projects from GitHub. We find that: 1) unit tested DL projects have positive correlation with the open-source project metrics and have a higher acceptance rate of pull requests, 2) 68% of the sampled DL projects are not unit tested at all, 3) the layer and utilities (utils) of DL models have the most unit tests. Based on these findings and previous research outcomes, we built a mapping taxonomy between unit tests and faults in DL projects. We discuss the implications of our findings for developers and researchers and highlight the need for unit testing in open-source DL projects to ensure their reliability and stability. The study contributes to this community by raising awareness of the importance of unit testing in DL projects and encouraging further research in this area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑