无法调度的调度：在大规模上驯服黑盒LLM推理
Scheduling the Unschedulable: Taming Black-Box LLM Inference at Scale
AI总结：
本文提出了一种三层客户端分解方法，通过适应性DRR分配、可行集评分排序和过载控制，解决黑盒LLM推理调度问题，实现高完成率和准时率。
摘要：
当提交时间可以预测输出token数量（Gan等人，2026）时，客户端侧调度针对黑盒LLM API成为半clairvoyant：决策基于粗略的token先验，尽管提供方的内部细节仍然隐藏。我们将这一边界问题分解为三个可分离的关切：分配（通过自适应DRR进行类间共享）、排序（通过可行集评分进行类内排序）和过载控制（在成本阶梯上显式接受/延迟/拒绝）。信息阶梯实验表明，粗略的量级先验——而不是仅类标签——是实用的客户端控制阈值；去除量级会使短请求的P95增加高达5.8倍，并降低截止时间满足率。在平衡/高拥堵情况下，完整堆栈实现了100%完成率、100%截止时间满足率和有用的goodput为4.2±1.6 SLO满足请求/秒，短P95在几十毫秒内处于配额分层隔离内。预测器噪声扫描确认在最多60%的乘法误差下能够优雅降级。重负载主导的领域将策略分开在完成、尾部和可解释性舍弃上。我们进一步比较了短优先级分配（偏向于交互式流量）与公平排队（在类间进行轮询）：公平排队在FIFO基础上实现了+32%的短请求P90改进，仅需+17%的长请求开销，而短优先级的+27% / +116%的权衡——证明了分配层可以适应不同的公平性目标而无需改变剩余堆栈。我们贡献了三层客户端分解、跨领域联合指标的受控评估、分配策略替代方案以及过载策略证据，将成本阶梯舍弃与陈述的服务目标联系起来。
英文摘要：
When output token counts can be predicted at submission time (Gan et al., 2026), client-side scheduling against a black-box LLM API becomes semi-clairvoyant: decisions condition on coarse token priors even though the provider's internals remain hidden. We decompose this boundary problem into three separable concerns: allocation (inter-class share via adaptive DRR), ordering (intra-class sequencing with feasible-set scoring), and overload control (explicit admit/defer/reject on a cost ladder). An information ladder experiment shows that coarse magnitude priors -- not class labels alone -- are the practical threshold for useful client control; removing magnitude inflates short-request P95 by up to $5.8\times$ and degrades deadline satisfaction. Under balanced / high congestion the full stack achieves 100% completion, 100% deadline satisfaction, and useful goodput of $4.2 \pm 1.6$ SLO-meeting requests/s with short P95 within tens of milliseconds of quota-tiered isolation. A predictor-noise sweep confirms graceful degradation under up to 60% multiplicative error. Heavy-dominated regimes separate policies on completion, tail, and interpretable shedding. We further compare short-priority allocation (biased toward interactive traffic) with Fair Queuing (round-robin across classes): Fair Queuing achieves +32% short-request P90 improvement over FIFO with only +17% long-request overhead, versus Short-Priority's +27% / +116% trade-off -- demonstrating that the allocation layer accommodates different fairness objectives without changing the remaining stack. We contribute the three-layer client-side decomposition, controlled evaluation of joint metrics across regimes, allocation-policy alternatives, and overload-policy evidence linking cost-ladder shedding to the stated service objective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
