arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.15705cs.DCcs.LG
收藏

动态重新分组以实现高效的早退推理

Batched Early-Exit LLM Inference Done Right with DREX

University of Pennsylvania(宾夕法尼亚大学);Microsoft Research(微软研究院)

Xuting Liu, Daniel Alexander, Jiaheng Lu, Siva Kesava Reddy Kakarla, Behnaz Arzani, Vincent Liu

AI总结:

DREX通过动态重新分组和优化策略提升早退推理效率,提高吞吐量并保障输出质量。

摘要:

早退(EE)是一种大型语言模型(LLM)架构,通过仅使用模型部分层生成更容易的token来加速推理。然而,传统批处理框架不适合EE LLMs,因为一批请求中并非所有请求都可能在同一时间准备好退出。现有解决方案要么强制对批次做出统一决定,从而忽略了EE机会,要么通过强制提前退出而降低输出质量。我们提出了动态重新分组,一种在每个早退点动态重新组织批次的解决方案。满足退出标准的请求立即被处理,而继续的请求则被保留在缓冲区中,重新分组为新批次并转发到更深层的层。我们引入了DREX,一种实现动态重新分组的早退推理系统,具有两个关键优化:1)无拷贝的重新分组缓冲区,避免物理数据移动;2)一个考虑EE和SLA的调度器,能够分析预测给定的重新分组操作是否具有盈利性。DREX还通过内存高效的态拷贝处理被跳过的层的缺失KV缓存。我们的评估显示,与基线方法相比,DREX在保持输出质量的同时,将吞吐量提高了2-12%。至关重要的是,DREX完全消除了不必要的退出,为保持早退模型预期的输出质量提供了关键保证。

英文摘要:

Early-exit (EE) LLMs let confident tokens skip a model's deep layers, but batched serving loses most of this gain. Current systems either force tokens to exit before their own ramp is confident, an error the ramp does not bound, or save almost nothing when decoding is weight-bound, because any continuing token makes a step read the deep weights. We present DREX, a batched EE serving system that never forces a token to exit. DREX introduces cross-iteration batching: tokens that continue past the exit layer wait there, and every few steps one deep pass over all waiting tokens reads the deep weights once for several steps. Its two policies differ in what such a token's request does: holding pauses it until the deep pass, and speculative holding lets it keep decoding from the exit layer's own guess, which the next deep pass verifies. Exited tokens are trusted and never verified. Using a calibrated cost model, a controller picks speculative holding where reading weights dominates a step and guesses are often right, and holding elsewhere. DREX keeps vLLM's asynchronous scheduling by planning each deep pass from exit counts of earlier steps. With synthetic exits where 90% of tokens exit at a quarter of the depth and 40% of guesses are correct, DREX achieves 2.22-2.24$\times$ decode speedup over non-EE across batch sizes on Llama-2-70B, while per-row skipping (vSkipper's design) gains at most 7.0%. On LayerSkip models, DREX adds no statistically resolved quality loss to the EE model's own. On CNN/DM at full memory, DREX with LayerSkip achieves 1.58-1.91$\times$ decode speedup while remaining within 1.4 ROUGE-2 points of the non-EE model. These speedups are 9-63% larger than EAGLE's across the evaluated model sizes, with each method measured against its own non-EE target.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑