Amphista:用于加速LLM推理的双向多头解码
Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference
AI总结:
Amphista通过双向注意力和分阶段适应层改进Medusa投机解码,在Vicuna 33B上实现最高2.75倍推理加速且保持生成质量。
摘要:
大型语言模型(LLMs)本质上采用自回归解码,在推理过程中缺乏并行性,导致推理速度显著缓慢。尽管Medusa等方法构建了并行化的头,但它们在跨不同预测位置之间缺乏足够的信息交互。为了克服这一限制,我们提出了Amphista,一个基于Medusa构建的增强型投机解码框架。具体来说,Amphista建模了一个能够并行推理的Auto-embedding Block,并引入双向注意力以实现不同草稿头之间的交互。此外,Amphista集成了Staged Adaptation Layers,确保语义信息从目标模型的自回归推理到草稿头的非自回归推理的无缝过渡,有效实现范式转换和特征融合。在Vicuna模型上使用MT-Bench和Spec-Bench进行的实验结果表明,Amphista在保持生成质量的同时实现了显著加速。在MT-Bench上,按墙钟时间计算,Amphista在Vicuna 33B上相比普通自回归解码实现了高达2.75$\ imes$的加速,相比Medusa实现了1.40$\ imes$的加速。
英文摘要:
Large Language Models (LLMs) inherently use autoregressive decoding, which lacks parallelism in inference and results in significantly slow inference speed. While methods such as Medusa constructs parallelized heads, they lack adequate information interaction across different prediction positions. To overcome this limitation, we introduce Amphista, an enhanced speculative decoding framework that builds upon Medusa. Specifically, Amphista models an Auto-embedding Block capable of parallel inference, incorporating bi-directional attention to enable interaction between different drafting heads. Additionally, Amphista integrates Staged Adaptation Layers, which ensure a seamless transition of semantic information from the target model's autoregressive inference to the drafting heads' non-autoregressive inference, effectively achieving paradigm shift and feature fusion. Experimental results on Vicuna models using MT-Bench and Spec-Bench demonstrate that Amphista achieves substantial acceleration while maintaining generation quality. On MT-Bench, Amphista delivers up to 2.75$\times$ speedup over vanilla autoregressive decoding and 1.40$\times$ over Medusa on Vicuna 33B in wall-clock time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。