arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.28175cs.LG
收藏

Strait: 机器学习推理服务中的优先级感知与干扰处理

Strait: Perceiving Priority and Interference in ML Inference Serving

Inria \& Sorbonne University Paris France;Inria Paris France;Inria \& Sorbonne University;Inria

Haidong Zhao, Nikolaos Georgantas

AI总结:

Strait系统通过优先级感知调度和干扰预测,提升高优先级任务的截止期限满足率,降低低优先级任务的开销。

摘要:

机器学习(ML)推理服务系统托管深度神经网络(DNN)模型,并在部署的GPU上调度 incoming 推理请求。然而,有限的任务优先级支持和并发执行下不充分的延迟估计可能限制其在本地场景中的应用。我们提出了Strait,一个旨在增强高GPU利用率下双优先级推理流量截止期限满足的的服务系统。为提高延迟估计,Strait模型数据传输期间的潜在竞争,并通过自适应预测模型考虑内核执行干扰。通过利用这些预测,它执行优先级感知调度以实现差异化处理。在高强度负载下的评估结果表明,Strait在高优先级任务中将截止期限违规率减少1.02至11.18个百分点,同时在低优先级任务上产生可接受的开销。与软件定义抢占方法相比,Strait还表现出更公平的性能。

英文摘要:

Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios. We present \emph{Strait}, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfer and accounts for kernel execution interference through an adaptive prediction model. By drawing on these predictions, it performs priority-aware scheduling to deliver differentiated handling. Evaluation results under intense workloads suggest that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on low-priority tasks. Compared to software-defined preemption approaches, Strait also exhibits more equitable performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑