不要想当然：比较开源库的软件开发工作量评估
Do Not Take It for Granted: Comparing Open-Source Libraries for Software Development Effort Estimation
摘要：
在过去的二十年里，几个机器学习（ML）库已经变得免费可用。许多研究已经使用这样的库进行预测软件工程（SE）任务的实证调查。然而，使用一个库而不是另一个库所产生的差异被忽略了，隐含地假设使用这些库中的任何一个都会为用户提供相同或非常相似的结果。本文旨在提高人们对使用不同ML库进行软件开发工作量估计（SEE）时所产生的差异的认识，SEE是研究最广泛的SE预测任务之一。为此，我们研究了4个确定性机器学习器，它们由3个用不同语言编写的最流行的ML开源库（即Scikit-Learn，Caret和Weka）提供。我们进行了一项彻底的实证研究，比较了机器学习器在两种最常见的SEE场景（即，out-of-the-box-ml和tuned-ml）以及对其API的文档和代码的深入分析。我们的研究结果表明，在总共研究的105个案例中，平均95%的案例中，3个库提供的预测结果不同。在大多数情况下，这些差异非常大，并产生高达约的错误估计。每个项目3,000小时。此外，我们的API分析表明，这些库为用户提供了不同级别的参数控制，并且总体上缺乏清晰度和一致性，这可能会误导用户。我们的研究结果强调，ML库是SEE研究的重要设计选择，这可能导致性能差异。然而，这种差异没有得到充分的记录。最后，我们强调开放的挑战，为图书馆的开发人员以及使用它们的研究人员和从业人员的建议。
英文摘要：
In the past two decades, several Machine Learning (ML) libraries have become freely available. Many studies have used such libraries to carry out empirical investigations on predictive Software Engineering (SE) tasks. However, the differences stemming from using one library over another have been overlooked, implicitly assuming that using any of these libraries would provide the user with the same or very similar results. This paper aims at raising awareness of the differences incurred when using different ML libraries for software development effort estimation (SEE), one of most widely studied SE prediction tasks. To this end, we investigate 4 deterministic machine learners as provided by 3 of the most popular ML open-source libraries written in different languages (namely, Scikit-Learn, Caret and Weka). We carry out a thorough empirical study comparing the performance of the machine learners on 5 SEE datasets in the two most common SEE scenarios (i.e., out-of-the-box-ml and tuned-ml) as well as an in-depth analysis of the documentation and code of their APIs. The results of our study reveal that the predictions provided by the 3 libraries differ in 95% of the cases on average across a total of 105 cases studied. These differences are significantly large in most cases and yield misestimations of up to approx. 3,000 hours per project. Moreover, our API analysis reveals that these libraries provide the user with different levels of control on the parameters one can manipulate, and a lack of clarity and consistency, overall, which might mislead users. Our findings highlight that the ML library is an important design choice for SEE studies, which can lead to a difference in performance. However, such a difference is under-documented. We conclude by highlighting open-challenges with suggestions for the developers of libraries as well as for the researchers and practitioners using them.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
