arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.21365cs.LGcs.CL
收藏

TIDE:基于令牌的深度执行用于每令牌早退的LLM推理

TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference

RightNow AI

Jaber Jaber, Osama Jaber

AI总结:

TIDE通过在周期性检查点层添加小型学习路由器,在推理时为每个令牌选择最早收敛的隐藏状态层,实现令牌早退,提升推理效率。

摘要:

大型语言模型无论难度如何都会将每个令牌传递给每一层。我们提出了TIDE,一种训练后系统,它在周期性检查点层附加了微小的学习路由器,并在推理时为每个令牌选择最早隐藏状态收敛的层。TIDE不需要模型重新训练,适用于任何HuggingFace因果语言模型,自动检测GPU架构,并通过融合的CUDA内核支持float32、float16和bfloat16。在NVIDIA A100上的DeepSeek R1 Distill 8B上,TIDE实现了100%的prefill早退率(5%的令牌在第11层早退,其余在第31层),将prefill延迟减少了7.2%,并提高了单批吞吐量6.6%。在自回归解码过程中,98-99%的令牌在模型正确解决多步骤数学问题时早退,该问题有95个唯一输出令牌。在Qwen3 8B(36层)上,批大小为8时吞吐量提高了8.1%。在2000个WikiText样本上校准耗时不到3分钟,生成约4MB的路由检查点。系统包含1308行Python和1081行CUDA/C++代码,通过74个测试。代码:https://github.com/RightNow-AI/TIDE

英文摘要:

Large language models run every token through every layer, regardless of difficulty. We present TIDE, a post-training system that attaches tiny learned routers at periodic checkpoint layers and, at inference time, selects the earliest layer whose hidden state has converged for each token. TIDE requires no model retraining, works with any HuggingFace causal LM, auto-detects GPU architecture, and supports float32, float16, and bfloat16 through fused CUDA kernels. On an NVIDIA A100 with DeepSeek R1 Distill 8B, TIDE achieves 100% prefill exit rate (5% of tokens exit at layer 11, the remaining at layer 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%. During autoregressive decoding, 98-99% of tokens exit early while the model correctly solves a multi-step math problem with 95 unique output tokens. On Qwen3 8B (36 layers), throughput improves by 8.1% at batch size 8. Calibration on 2,000 WikiText samples takes under 3 minutes and produces a ~4 MB router checkpoint. The system comprises 1,308 lines of Python and 1,081 lines of CUDA/C++ with 74 passing tests. Code: https://github.com/RightNow-AI/TIDE

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑