EasySpec: 用于高效多GPU利用的分层并行推测解码
EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
AI总结：
EasySpec通过分层并行推测策略提升多GPU推理效率，实现4.17倍速度提升，同时保持模型准确性。
摘要：
推测解码是一种有效的、无损的大型语言模型（LLM）推理加速方法。它使用较小的模型生成一个草稿令牌序列，然后由原始基础模型进行验证。在多GPU系统中，通过张量并行（TP）可以进一步减少推理延迟，但草稿模型的最佳TP大小通常小于基础模型，导致在草稿阶段出现GPU空闲。我们发现这种低效性源于层间执行的顺序性，这看似自然但实际上是不必要的。因此，我们提出了EasySpec，一种分层并行推测策略，以优化多GPU利用的效率。EasySpec打破了草稿模型中的层间数据依赖性，使多个层能够在多个设备上同时运行作为'模糊'推测。在每次草稿-验证迭代后，草稿模型的关键值缓存通过一次前向传递进行校准，从而在极小的额外延迟下防止长期模糊误差积累。EasySpec是一种无需训练且可插拔的方法。我们在几种主流开源LLM上评估了EasySpec，使用同一系列中较小的模型作为草稿模型。结果表明，EasySpec相比传统解码可实现峰值加速4.17倍，同时保持基础LLM的原始分布。具体而言，草稿阶段可加速高达1.62倍，最大推测准确率下降仅7%。代码可在https://github.com/Yize-Wu/EasySpec获取。
英文摘要：
Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which is then verified by the original base model. In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage. We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary. Therefore, we propose EasySpec, a layer-parallel speculation strategy that optimizes the efficiency of multi-GPU utilization. EasySpec breaks the inter-layer data dependencies in the draft model, enabling multiple layers to run simultaneously across multiple devices as 'fuzzy' speculation. After each drafting-and-verification iteration, the draft model's key-value cache is calibrated in a single forward pass, preventing long-term fuzzy-error accumulation at minimal additional latency. EasySpec is a training-free and plug-in method. We evaluated EasySpec on several mainstream open-source LLMs, using smaller versions of models from the same series as drafters. The results demonstrate that EasySpec can achieve a peak speedup of 4.17x compared to vanilla decoding, while preserving the original distributions of the base LLMs. Specifically, the drafting stage can be accelerated by up to 1.62x with a maximum speculation accuracy drop of only 7%. The code is available at https://github.com/Yize-Wu/EasySpec.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
