arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.34103cs.CLcs.AIcs.LG
收藏

T2D:面向高效推测解码的目标到草稿模型剪枝与适配

T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding

Bellini College of AI, Cybersecurity, and Computing, University of South Florida(南佛罗里达大学贝利尼人工智能、网络安全与计算学院);School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);University of California, Davis(加州大学戴维斯分校)

Shahriar Kabir Nahin, Xiao Ye, Muhao Chen, Ben Zhou, Anshuman Chhabra

AI总结:

该研究针对现有推测解码方法的权衡缺陷,提出T2D框架,通过感知延迟的结构化剪枝与知识蒸馏从目标模型构建草稿模型,可剪枝目标模型达92%,且支持即插即用,提升推测解码性能。

摘要:

推测解码可显著加速大型语言模型(LLM)的自回归文本生成,但其效能很大程度上依赖两个关键因素:草稿模型与目标模型的分布对齐,以及推理时草稿模型的延迟。然而,现有方法通常使用独立预训练的小型草稿模型,或对目标模型进行手动选择架构的剪枝,因此未能明确平衡草稿模型对齐与令牌生成效率之间的权衡。为解决这一差距,我们将草稿模型设计建模为优化问题,并提出T2D框架,该框架利用感知延迟的结构化剪枝和知识蒸馏,直接从目标模型构建草稿模型。通过这种方式,T2D获得的草稿模型在实测推理延迟和目标模型响应的交叉熵损失上均经过联合优化,从而实现卓越的推测解码性能。我们在多领域推测解码基准以及多个不同规模的目标模型家族上进行了大量实验,结果表明,与竞争基线相比,T2D可剪枝目标模型多达92%,以获得高性能的草稿模型。此外,通过T2D获得的草稿模型还可被其他推测解码令牌对齐和知识蒸馏方法以即插即用的方式使用,从而进一步提升性能。我们的研究结果强调,联合探索草稿模型架构-对齐边界对于释放LLM推测解码的全部潜力至关重要。

英文摘要:

Speculative decoding significantly accelerates autoregressive text generation in Large Language Models (LLMs), but its efficacy depends largely on two key factors: the distributional alignment between the draft and the target model, and the draft model's latency during inference. However, existing approaches typically utilize independently pretrained small draft models or seek to prune the target model to a manually chosen architecture, thereby failing to explicitly balance the trade-off between draft model alignment and token generation efficiency. To address this gap, we formulate draft model design as an optimization problem and propose T2D, a framework that constructs draft models directly from the target model using latency-aware structured pruning and knowledge distillation. In this manner, T2D obtains draft models that are jointly optimized for both measured inference latency and cross-entropy loss on the target model's responses, resulting in superlative speculative decoding performance. Through extensive experiments on multi-domain speculative decoding benchmarks and across several target model families of differing scale, we demonstrate how T2D prunes up to 92\% of the target model to obtain highly performant draft models compared to competitive baselines. Moreover, draft models obtained via T2D can also be utilized by other speculative decoding token-alignment and knowledge distillation methods in a plug-and-play fashion, leading to further gains. Our findings underscore the importance of jointly navigating the draft model architecture-alignment frontier to unlock the full potential of speculative decoding for LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑