软件性能预测:没有银弹
Predicting Performance of Software Configurations: There is no Silver Bullet
摘要:
许多软件系统提供配置选项来定制它们的功能和非功能属性(例如,性能)。通常,用户对(性能)最佳配置感兴趣,但由于缺少关于各个配置选项及其交互的影响的信息,因此很难找到它。在过去,各种监督机器学习技术已被用于预测所有配置的性能,并确定最佳配置。在文献中,有大量的机器学习技术和采样策略可供选择。不过,目前还不清楚它们在多大程度上影响预测准确性。我们进行了一项比较研究,在预测所有配置的性能时,考虑6种机器学习技术,18种采样策略和6种主题软件系统的平均预测精度。我们发现,学习技术和采样策略对预测精度有很大的影响。我们进一步观察到,一些学习技术(例如,随机森林)优于其它学习技术(例如,k-最近邻)。此外,由于预测准确度在很大程度上取决于主题系统,因此考虑到准确性和测量开销之间的权衡,不存在在所有情况下都是最佳的学习技术和采样策略的组合,这符合著名的无免费午餐定理。
英文摘要:
Many software systems offer configuration options to tailor their functionality and non-functional properties (e.g., performance). Often, users are interested in the (performance-)optimal configuration, but struggle to find it, due to missing information on influences of individual configuration options and their interactions. In the past, various supervised machine-learning techniques have been used to predict the performance of all configurations and to identify the optimal one. In the literature, there is a large number of machine-learning techniques and sampling strategies to select from. It is unclear, though, to what extent they affect prediction accuracy. We have conducted a comparative study regarding the mean prediction accuracy when predicting the performance of all configurations considering 6 machine-learning techniques, 18 sampling strategies, and 6 subject software systems. We found that both the learning technique and the sampling strategy have a strong influence on prediction accuracy. We further observed that some learning techniques (e.g., random forests) outperform other learning techniques (e.g., k-nearest neighbor) in most cases. Moreover, as the prediction accuracy strongly depends on the subject system, there is no combination of a learning technique and sampling strategy that is optimal in all cases, considering the tradeoff between accuracy and measurement overhead, which is in line with the famous no-free-lunch theorem.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。