发表机构
Concordia University; University of L’Aquila; SAP; Oracle; University of Würzburg; Charles University; University of Hamburg（康考迪亚大学; 拉奎拉大学; 思爱普; 甲骨文; 维尔茨堡大学; 查理大学; 汉堡大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文评估十二种变点检测方法在真实软件性能数据上的准确性，揭示其在性能回归分析中的适用性与有效性。
AI 中文摘要
软件系统中的性能问题是一个关键的质量问题，它可能侵蚀用户信任、违反服务级别协议，并最终影响业务效率。因此，软件性能工程已将重点转向开发稳健的技术，以便在开发周期中尽早检测性能回归。性能回归分析通常依赖于性能测量的时间序列来检测性能行为中的显著变化。变点检测（CPD）方法已被广泛用于自动化识别各个领域（包括金融、医疗保健和性能监控）中的此类变化。然而，这些方法在软件性能测量中的有效性尚未得到彻底评估。在本文中，我们提出了一项综合研究，以评估各种CPD方法在真实世界软件性能测量数据集上的有效性。我们首先从三个大型软件系统中收集性能测量数据，并刻画性能时间序列数据的独特属性。然后，我们进行大规模工作，注释并评估人类注释者识别潜在性能变化的一致性。此后，我们评估了十二种不同CPD方法在检测潜在性能变化方面的准确性，为它们在软件性能回归分析中的适用性和有效性提供了见解。
英文摘要
Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.
