arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.11273cs.LG
收藏

具有持续时间感知的ASR服务调度

Duration Aware Scheduling for ASR Serving Under Workload Drift

Sprinklr

Darshan Makwana, Yash Jogi, Harsh Kotta, Aayush Kubba

AI总结:

本文提出基于持续时间的ASR服务调度方法,通过SJF和HRRN算法在负载漂移下减少E2E延迟并控制尾部延迟,提升服务性能。

摘要:

在大规模自动语音识别(ASR)服务管道中,调度策略在确定端到端(E2E)延迟方面起着关键作用。然而,广泛使用的服务引擎依赖于先进先出(FCFS)调度,这忽略了请求持续时间的变化,导致在负载漂移下出现头部阻塞。我们展示了音频持续时间是ASR模型(如Whisper)中作业处理时间的准确代理,并利用这一见解实现了持续时间感知调度。我们将两种经典算法,最短作业优先(SJF)和最高响应比下一个(HRRN),整合到vLLM中,并在现实和漂移的负载下评估它们。在LibriSpeech测试清洁数据集上,与基线相比,SJF在高负载下将中位E2E延迟降低了高达73%,但由于长请求被饿死,90百分位尾部延迟增加了高达97%。HRRN解决了这一权衡:它将中位E2E延迟降低了高达28%,同时将尾部延迟退化限制在最多24%以内。这些收益在负载漂移下持续存在,没有吞吐量惩罚,且每个请求的调度开销小于0.1毫秒。

英文摘要:

Scheduling policies in large-scale Automatic Speech Recognition (ASR) serving pipelines play a key role in determining end-to-end (E2E) latency. Yet, widely used serving engines rely on first-come-first-served (FCFS) scheduling, which ignores variability in request duration and leads to head-of-line blocking under workload drift. We show that audio duration is an accurate proxy for job processing time in ASR models such as Whisper, and use this insight to enable duration-aware scheduling. We integrate two classical algorithms, Shortest Job First (SJF) and Highest Response Ratio Next (HRRN), into vLLM and evaluate them under realistic and drifted workloads. On LibriSpeech test-clean, compared to baseline, SJF reduces median E2E latency by up to $73\%$ at high load, but increases $90$th-percentile tail latency by up to $97\%$ due to starvation of long requests. HRRN addresses this trade-off: it reduces median E2E latency by up to $28\%$ while bounding tail-latency degradation to at most $24\%$. These gains persist under workload drift, with no throughput penalty and $<0.1$\,ms scheduling overhead per request.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑