LatencyPrism: 一种在线非侵入式延迟塑形系统以保证LLM推理的SLA
LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference
AI总结:
LatencyPrism是一种非侵入式延迟塑形系统,通过实时监控和异常检测保证LLM推理的SLA要求。
摘要:
LLM推理延迟严重决定用户体验和运营成本,直接影响在SLA约束下的吞吐量。即使短暂的延迟峰值也会降低服务质量,尽管平均性能是可接受的。然而,分布式推理环境结合了多样化的软件框架和XPU架构以及动态负载,使延迟分析变得具有挑战性。受侵入式设计的限制,需要服务重启甚至暂停,以及受硬件限制的实现无法适应异构推理环境,现有的AI分析方法往往无法满足实时生产分析的需求。我们提出了LatencyPrism,这是首个零侵入多平台延迟塑形系统。它旨在将推理延迟分解到流水线中,主动检测推理延迟异常,并保证遵守SLA,而无需修改代码或服务重启。LatencyPrism已在数千个XPU上部署了六个月以上。它在批次级别实现了低开销的实时监控,警报可在毫秒内触发。这种方法在F1得分为0.98的情况下区分了负载驱动的延迟变化和指示潜在问题的异常。我们还进行了广泛的实验和调查,以展示LatencyPrism的能力。此外,我们引入了首个LLM异常模拟工具包,以促进未来在鲁棒和可预测的推理系统中的研究。
英文摘要:
LLM inference latency critically determines user experience and operational costs, directly impacting throughput under SLO constraints. Even brief latency spikes degrade service quality despite acceptable average performance. However, distributed inference environments featuring diverse software frameworks and XPU architectures combined with dynamic workloads make latency analysis challenging. Constrained by intrusive designs that necessitate service restarts or even suspension, and by hardware-bound implementations that fail to adapt to heterogeneous inference environments, existing AI profiling methods are often inadequate for real-time production analysis. We present LatencyPrism, the first zero-intrusion multi-platform latency sculpting system. It aims to break down the inference latency across pipeline, proactively alert on inference latency anomalies, and guarantee adherence to SLOs, all without requiring code modifications or service restarts. LatencyPrism has been deployed across thousands of XPUs for over six months. It enables low-overhead real-time monitoring at batch level with alerts triggered in milliseconds. This approach distinguishes between workload-driven latency variations and anomalies indicating underlying issues with an F1-score of 0.98. We also conduct extensive experiments and investigations into root cause analysis to demonstrate LatencyPrism's capability. Furthermore, we introduce the first LLM anomaly simulation toolkit to facilitate future research in robust and predictable inference systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。