arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2003.00584cs.SEcs.DBcs.PF
收藏

软件性能测试中的变点检测

Change Point Detection in Software Performance Testing

David Daly, William Brown, Henrik Ingo, Jim O'Leary, David Bradford

摘要:

我们描述了我们的过程中自动检测的性能变化的软件产品中存在的噪声。当我们的软件产品的更改提交到我们的源代码库时,会定期运行大量的测试,我们希望确定导致性能退化的提交。以前,我们依赖于手动检查时间序列图来识别重大变化。后来被基于阈值的检测系统所取代,但这两个系统都不足以及时发现性能的变化。这项工作描述了我们最近实现的一个变化点检测系统的E-分裂的手段算法。该算法产生一系列变化点,这些变化点表示从给定的性能结果历史的显著变化。一个人检查可操作的更改的更改点列表,然后对这些更改进行分类以进行进一步检查。使用变化点检测对我们检测性能变化的能力产生了巨大的影响。在数量上,它大大降低了我们对性能变化的误报率,而在质量上,它使整个性能评估过程更容易,更有成效(例如,更小的回归),更及时。

英文摘要:

We describe our process for automatic detection of performance changes for a software product in the presence of noise. A large collection of tests run periodically as changes to our software product are committed to our source repository, and we would like to identify the commits responsible for performance regressions. Previously, we relied on manual inspection of time series graphs to identify significant changes. That was later replaced with a threshold-based detection system, but neither system was sufficient for finding changes in performance in a timely manner. This work describes our recent implementation of a change point detection system built upon the E-Divisive means algorithm. The algorithm produces a list of change points representing significant changes from a given history of performance results. A human reviews the list of change points for actionable changes, which are then triaged for further inspection. Using change point detection has had a dramatic impact on our ability to detect performance changes. Quantitatively, it has dramatically dropped our false positive rate for performance changes, while qualitatively it has made the entire performance evaluation process easier, more productive (ex. catching smaller regressions), and more timely.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑