双人乐器:一种改进云服务基准测试灵敏度的代理方法
Duet instrumentation: An Agentic Approach to Improving Sensitivity in Cloud Service Benchmarking
AI总结:
本文提出了一种名为duet instrumentation的新基准测试方法,利用大型语言模型的代码理解能力,通过分析两个连续应用版本之间的代码变更,提高云服务基准测试的灵敏度。
摘要:
持续的云服务性能基准测试对于在部署到生产环境之前尽早检测性能问题至关重要。然而,使用应用程序基准测试来检测性能回归具有挑战性,因为这些测试通常将被测系统视为黑盒,而变量的I/O调用或底层云基础设施性能特征的变化会增加难度。微基准测试通常更敏感和准确,但实现和运行它们也更耗时。此外,它们不捕捉整个集成系统的性能。因此,全面的性能评估通常需要结合两种方法。为了解决应用程序基准测试的不足,我们提出了duet instrumentation,一种由最近在大型语言模型(LLM)代码理解方面的进展所支持的新基准测试范式。该方法的思路是分析两个连续应用版本之间的代码变更,并在同步基准测试两个应用版本时,直接测量与性能相关的变化,从而以更高的灵敏度揭示性能变化。我们设计了一个系统,能够可靠地自动化评估和仪器化两个应用版本之间性能相关的代码变更。在使用现实测试环境应用程序进行测试时,我们发现我们的原型在使用行距离阈值为五时,与理想仪器化相比,实现了58%的精度、93%的召回率和71%的特异性(平均跨任务)。在下游应用基准测试中,我们发现我们的原型能够检测性能回归,其注入严重性比传统duet应用基准测试低至5倍,同时保持相似的A/A延迟分布。
英文摘要:
Continuous cloud service performance benchmarking is essential for detecting performance bugs early before deploying them to production. However, detecting performance regressions using application benchmarks, which usually treat the system under test as a black box, is challenging due to variable I/O calls or changing performance characteristics of the underlying cloud infrastructure. Microbenchmarks are often more sensitive and accurate, but also more time-consuming to implement and run. Further, they do not capture the performance of the integrated system as a whole. A comprehensive performance assessment therefore typically requires a combination of both approaches. To address the shortcomings of application benchmarks, we propose duet instrumentation, a novel benchmarking paradigm enabled by recent advancements in large language model (LLM) code understanding. The idea is to analyze code changes between two consecutive application versions and measure performance differences directly at performance-relevant changes during a synchronized benchmark of both application versions, uncovering performance changes with higher sensitivity. We design a system that reliably automates the assessment and instrumentation of performance-relevant code changes between the two application versions. In experiments with a realistic testbed application offering configurable performance regressions, we find that our prototype achieves 58% precision, 93% recall, and 71% specificity (averaged across tasks) when comparing the generated instrumentation against the ideal instrumentation with a line-distance threshold of five. In the downstream application benchmark, we find that our prototype can detect performance regressions at up to 5x lower injected severity compared to a traditional duet application benchmark while preserving similar A/A latency distributions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。