GliDe with a CaPE：一种低负担的投机解码加速方法
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
AI总结：
本研究提出GliDe和CaPE两种低负担修改，通过重用缓存和置信度选择，将投机解码加速提升至2.61倍，显著降低大语言模型解码延迟。
摘要：
投机解码是一种相对较新的解码框架，利用小型高效的草稿模型来减少大语言模型的延迟。在本研究中，我们引入了GliDe和CaPE，这两种对标准投机解码的低负担修改，旨在进一步提高冻结大语言模型的解码速度。具体来说，GliDe是一种修改后的草稿模型架构，它重用目标大语言模型的缓存键和值，而CaPE是一种提议扩展方法，利用草稿模型的置信度分数来帮助选择额外的候选令牌以供验证。在不同基准上的大量实验表明，我们提出的GliDe草稿模型显著减少了预期的解码延迟。使用墙钟时间的额外评估显示，GliDe可以将Vicuna模型的加速提升至2.17倍，并通过CaPE进一步将改进扩展到2.61倍。我们将发布我们的代码、数据和训练好的草稿模型。
英文摘要：
Speculative decoding is a relatively new decoding framework that leverages small and efficient draft models to reduce the latency of LLMs. In this study, we introduce GliDe and CaPE, two low-hassle modifications to vanilla speculative decoding to further improve the decoding speed of a frozen LLM. Specifically, GliDe is a modified draft model architecture that reuses the cached keys and values from the target LLM, while CaPE is a proposal expansion method that uses the draft model's confidence scores to help select additional candidate tokens for verification. Extensive experiments on different benchmarks demonstrate that our proposed GliDe draft model significantly reduces the expected decoding latency. Additional evaluation using walltime reveals that GliDe can accelerate Vicuna models up to 2.17x and further extend the improvement to 2.61x with CaPE. We will release our code, data, and the trained draft models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
