从令牌到层：重新定义MoE服务中的无停滞调度
From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill
AI总结：
本文提出分层预填调度方法，通过将Transformer层组作为调度单元，减少MoE模型中的专家权重重复加载，提升TTFT和TBT性能，降低能耗和带宽需求。
摘要：
大型语言模型（LLM）生产推理必须同时满足时间到第一个令牌（TTFT）和时间间隔令牌（TBT）的严格服务级别目标，同时在固定计算、内存和互连预算下最大化吞吐量。现代服务系统采用无停滞调度技术，如分块预填，将长提示的处理沿令牌维度分割，并在正在进行的解码迭代中交错预填。虽然分块预填在稳定TBT方面有效，但在混合专家（MoE）模型中会带来显著的开销：冗余专家权重加载使内存流量增加高达39%并增加能耗。我们提出分层预填，一种新的调度范式，将Transformer层组作为主要调度单元，专门针对MoE服务。通过垂直划分模型为连续的层组，并在组间交错预填和解码，分层预填在保持无停滞解码的同时消除分块引起的MoE权重重新加载。它减少了离芯片带宽需求，降低了TTFT最高达70%，端到端延迟41%和每令牌能耗最高达22%。评估显示，分层预填在分块预填上持续改进TTFT-TBT帕累托前沿，减少专家负载流量和能耗，同时保持无停滞解码。总体而言，将调度轴从令牌转向层解锁了高效率、能源感知的MoE服务在共置环境中的新操作模式。
英文摘要：
Large Language Model (LLM) inference in production must meet stringent service-level objectives for both time-to-first-token (TTFT) and time-between-token (TBT) while maximizing throughput under fixed compute, memory, and interconnect budgets. Modern serving systems adopt stall-free scheduling techniques such as chunked prefill, which splits the processing of long prompts along the token dimension and interleaves prefill with ongoing decode iterations. While effective at stabilizing TBT, chunked prefill incurs substantial overhead in Mixture-of-Experts (MoE) models: redundant expert weight loads increase memory traffic by up to 39% and inflate energy consumption. We propose layered prefill, a new scheduling paradigm that treats transformer layer groups as the primary scheduling unit, specifically targeting MoE serving. By vertically partitioning the model into contiguous layer groups and interleaving prefill and decode across the groups, layered prefill sustains stall-free decoding while eliminating chunk-induced MoE weight reloads. It reduces off-chip bandwidth demand, lowering TTFT by up to 70%, end-to-end latency by 41% and per-token energy by up to 22%. Evaluations show that layered prefill consistently improves the TTFT--TBT Pareto frontier over chunked prefill, reducing expert-load traffic and energy cost while maintaining stall-free decoding. Overall, shifting the scheduling axis from tokens to layers unlocks a new operating regime for high-efficiency, energy-aware MoE serving in co-located environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
