arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.03486cs.LGcs.AI
收藏

基于推理的异常检测框架:一种跨领域异常检测的实时、可扩展且自动化的方法

Reasoning-based Anomaly Detection Framework: A Real-time, Scalable, and Automated Approach to Anomaly Detection Across Domains

Apple(苹果公司)

Anupam Panwar, Himadri Pal, Jiali Chen, Kyle Cho, Riddick Jiang, Miao Zhao, Rajiv Krishnamurthy

AI总结:

针对大型分布式系统异常检测中数据量大、数据集异构、根因定位难的问题,提出集成mSelect自动算法与超参数调优、检测后根因分析的RADF框架,实验显示其在多数公共基准数据集上AUC表现优于现有最优模型。

摘要:

在大型分布式系统中检测异常面临若干挑战。第一个挑战源于需要处理的数据量极为庞大。在高吞吐量环境中标记异常需要同时审慎考虑算法与系统设计。第二个挑战来自生产环境中使用该系统的时间序列数据集的异构性。在实践中,异常检测系统很少仅针对单个用例部署。通常需要监控多个指标,且这些指标往往横跨多个领域(例如工程、业务与运营)。一刀切的方法几乎行不通,因此这些系统需要针对每个应用进行微调——而这项工作通常由人工完成。第三个挑战在于,在这类场景中确定异常的根本原因无异于大海捞针。实时识别出与异常时间序列数据存在因果关联的时间序列数据集是一个非常困难的问题。\n在本文中,我们描述了一个可应对上述挑战的统一框架。基于推理的异常检测框架(Reasoning based Anomaly Detection Framework,RADF)旨在针对超大规模数据集执行实时异常检测。该框架采用了一种名为mSelect的新技术,可自动完成每个用例的算法选择与超参数调优流程。最后,它集成了检测后处理能力,可加快异常分流与根本原因确定的速度。我们的大量实验表明,在mSelect的加持下,RADF在9个公共基准数据集中的5个上,AUC表现优于当前最优的异常检测模型。RADF在9个数据集中的7个上取得了超过0.85的AUC,这一表现是其他任何现有最优模型都无法企及的。

英文摘要:

Detecting anomalies in large, distributed systems presents several challenges. The first challenge arises from the sheer volume of data that needs to be processed. Flagging anomalies in a high-throughput environment calls for a careful consideration of both algorithm and system design. The second challenge comes from the heterogeneity of time-series datasets that leverage such a system in production. In practice, anomaly detection systems are rarely deployed for a single use case. Typically, there are several metrics to monitor, often across several domains (e.g. engineering, business and operations). A one-size-fits-all approach rarely works, so these systems need to be fine-tuned for every application - this is often done manually. The third challenge comes from the fact that determining the root-cause of anomalies in such settings is akin to finding a needle in a haystack. Identifying (in real time) a time-series dataset that is associated causally with the anomalous time-series data is a very difficult problem. In this paper, we describe a unified framework that addresses these challenges. Reasoning based Anomaly Detection Framework (RADF) is designed to perform real time anomaly detection on very large datasets. This framework employs a novel technique (mSelect) that automates the process of algorithm selection and hyper-parameter tuning for each use case. Finally, it incorporates a post-detection capability that allows for faster triaging and root-cause determination. Our extensive experiments demonstrate that RADF, powered by mSelect, surpasses state-of-the-art anomaly detection models in AUC performance for 5 out of 9 public benchmarking datasets. RADF achieved an AUC of over 0.85 for 7 out of 9 datasets, a distinction unmatched by any other state-of-the-art model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑