昆仑异常排查器：面向大模型分布式推理的核级异常检测与因果推理
Kunlun Anomaly Troubleshooter: Enabling Kernel-Level Anomaly Detection and Causal Reasoning for Large Model Distributed Inference
AI总结：
本文提出首个面向大模型分布式推理的异常排查框架KAT，利用GPU函数跟踪数据实现纳秒级核级异常检测，并结合领域适配LLM进行因果推理，在生产环境中取得高精确率与召回率，显著提升排查效率。
摘要：
大模型分布式推理（LMDI）的异常排查仍然是一项关键挑战。在分布式系统中解决诸如推理性能下降或延迟抖动等异常，需要领域专家投入大量人工工作，导致诊断过程极为耗时且准确率相对较低。本文介绍了昆仑异常排查器（KAT），这是首个专为LMDI量身定制的异常排查框架。KAT通过两项核心创新来解决这一问题。第一，KAT利用GPU工作节点的同步性与一致性，创新性地利用函数跟踪数据，以纳秒级分辨率精确检测核级异常及相关硬件组件。第二，KAT将这些检测结果集成到领域适配的大语言模型（LLM）中，对复杂异常症状进行系统性的因果推理和自然语言解释。在阿里云服务生产环境中进行的评估表明，KAT在异常检测中实现了超过0.884的精确率和0.936的召回率，提供了详细的异常洞察，显著缩小了诊断范围，并提高了排查的效率和成功率。
英文摘要：
Anomaly troubleshooting for large model distributed inference (LMDI) remains a critical challenge. Resolving anomalies such as inference performance degradation or latency jitter in distributed system demands significant manual efforts from domain experts, resulting in extremely time-consuming diagnosis processes with relatively low accuracy. In this paper, we introduce Kunlun Anomaly Troubleshooter (KAT), the first anomaly troubleshooting framework tailored for LMDI. KAT addresses this problem through two core innovations. First, KAT exploits the synchronicity and consistency of GPU workers, innovatively leverages function trace data to precisely detect kernel-level anomalies and associated hardware components at nanosecond resolution. Second, KAT integrates these detection results into a domain-adapted LLM, delivering systematic causal reasoning and natural language interpretation of complex anomaly symptoms. Evaluations conducted in Alibaba Cloud Service production environment indicate that KAT achieves over 0.884 precision and 0.936 recall in anomaly detection, providing detail anomaly insights that significantly narrow down the diagnostic scope and improve both the efficiency and success rate of troubleshooting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
