arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.15988cs.SEcs.AI
收藏

配置编码对学习软件性能有影响吗?编码方案的实证研究

Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes

Loughborough University(拉夫堡大学)

Jingzhi Gong, Tao Chen

摘要:

学习和预测可配置软件系统的性能有助于提供更好的质量保证。其中一个重要的工程决策是如何将配置编码到构建的模型中。尽管存在不同的编码方案,但对于哪种方案更好以及在什么情况下更好仍然知之甚少,因为社区通常依赖于一些以特定方式告知决策的一般信念。为了弥合这一差距,在本文中,我们经验比较了广泛使用的编码方案的软件性能学习,即标签,缩放标签,和独热编码。该研究涵盖了五个系统,七个模型和三种编码方案,导致105个调查案例。我们的主要研究结果显示:(1)以逐个情况的方式进行试错以找到最佳编码方案可能是相当昂贵的,在某些模型和系统上需要高达400+小时;(2)独热编码通常导致最准确的结果,而缩放标签编码在不同模型上的准确性通常较弱;(3)相反,缩放标签编码倾向于导致跨模型/系统的最快训练时间,而独热编码是最慢的;(4)对于所有研究的模型,标签和缩放标签编码通常会导致准确性和训练时间之间的偏差相对较小,但配对模型会根据系统而有所不同。我们讨论了从我们的研究结果中得出的可操作的建议,希望为社区提供对这个主题的更好的理解。为了促进开放科学,这项工作的数据和代码可以在https://github.com/ideas-labo/MSR2022-encoding-study上公开访问。

英文摘要:

Learning and predicting the performance of a configurable software system helps to provide better quality assurance. One important engineering decision therein is how to encode the configuration into the model built. Despite the presence of different encoding schemes, there is still little understanding of which is better and under what circumstances, as the community often relies on some general beliefs that inform the decision in an ad-hoc manner. To bridge this gap, in this paper, we empirically compared the widely used encoding schemes for software performance learning, namely label, scaled label, and one-hot encoding. The study covers five systems, seven models, and three encoding schemes, leading to 105 cases of investigation. Our key findings reveal that: (1) conducting trial-and-error to find the best encoding scheme in a case by case manner can be rather expensive, requiring up to 400+ hours on some models and systems; (2) the one-hot encoding often leads to the most accurate results while the scaled label encoding is generally weak on accuracy over different models; (3) conversely, the scaled label encoding tends to result in the fastest training time across the models/systems while the one-hot encoding is the slowest; (4) for all models studied, label and scaled label encoding often lead to relatively less biased outcomes between accuracy and training time, but the paired model varies according to the system. We discuss the actionable suggestions derived from our findings, hoping to provide a better understanding of this topic for the community. To promote open science, the data and code of this work can be publicly accessed at https://github.com/ideas-labo/MSR2022-encoding-study.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑