描述和缓解工业云系统中的警报反模式
Characterizing and Mitigating Anti-patterns of Alerts in Industrial Cloud Systems
摘要:
警报对于在云异常时请求及时的人工干预至关重要。警报的质量会显著影响云的可靠性和云提供商的业务收入。在实践中,我们观察到,由于大量存在误导性、非信息性、不可操作的警报,待命工程师无法快速定位和修复故障云服务。我们将警报的无效性称为“警报的反模式”。为了更好地理解警报的反模式,并提供可操作的措施来减轻反模式,在本文中,我们进行了第一次实证研究的做法,减轻反模式的警报在工业云系统。我们研究了领先的云提供商华为云的警报策略和警报处理流程。我们的研究结合了两年内对数百万警报的定量分析和对18位经验丰富的工程师的调查。因此,我们总结了警报的四个单独的反模式和两个集体反模式。我们还总结了四个当前的反应,以减轻警报的反模式,和警报策略的配置的一般预防性指导方针。最后,我们建议探索自动评估的质量警报(QoA),包括指示性,精度和可重复性的警报,作为未来的研究方向,有助于自动检测警报的反模式。研究结果对优化云监控系统、提高云服务的可靠性具有一定的参考价值。
英文摘要:
Alerts are crucial for requesting prompt human intervention upon cloud anomalies. The quality of alerts significantly affects the cloud reliability and the cloud provider's business revenue. In practice, we observe on-call engineers being hindered from quickly locating and fixing faulty cloud services because of the vast existence of misleading, non-informative, non-actionable alerts. We call the ineffectiveness of alerts "anti-patterns of alerts". To better understand the anti-patterns of alerts and provide actionable measures to mitigate anti-patterns, in this paper, we conduct the first empirical study on the practices of mitigating anti-patterns of alerts in an industrial cloud system. We study the alert strategies and the alert processing procedure at Huawei Cloud, a leading cloud provider. Our study combines the quantitative analysis of millions of alerts in two years and a survey with eighteen experienced engineers. As a result, we summarized four individual anti-patterns and two collective anti-patterns of alerts. We also summarize four current reactions to mitigate the anti-patterns of alerts, and the general preventative guidelines for the configuration of alert strategy. Lastly, we propose to explore the automatic evaluation of the Quality of Alerts (QoA), including the indicativeness, precision, and handleability of alerts, as a future research direction that assists in the automatic detection of alerts' anti-patterns. The findings of our study are valuable for optimizing cloud monitoring systems and improving the reliability of cloud services.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。