arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.35957cs.LGcs.ITeess.SPmath.IT
收藏

基于通信高效投机解码的分布式大语言模型推理

Distributed LLM Inference via Communication-Efficient Speculative Decoding

School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院电气工程学院)

Joohyun Shin, Byeonghyeon Park, Keumseo Ryum, Joonhyuk Kang

AI总结:

针对分布式投机解码的词汇级通信开销问题,提出DISCO框架,通过引入SLM服务器端副本减少传输量,在多数据集实验中提升解码加速。

摘要:

分布式投机解码已成为设备-服务器协同大语言模型(LLM)推理的有前景方法,但其解码加速常受限于词汇级通信开销。现有分布式投机解码方法中,精确拒绝采样需在设备与服务器间传输LLM词汇的完整token分布,在带宽受限环境中成本高昂。为解决该问题,本文提出DISCO,一种通信高效的分布式投机解码框架,其消除了词汇级分布传输同时保留精确目标分布。DISCO通过引入草稿小语言模型(SLM)的服务器端副本实现,使服务器可本地获取拒绝采样所需的草稿侧分布。因此,DISCO仅在设备与服务器间传输草稿token索引和标量概率值。在多个数据集上的实验结果表明,DISCO相比现有分布式投机解码方法始终实现更高的解码加速,尤其在严格带宽约束下。

英文摘要:

Distributed speculative decoding has emerged as a promising approach for device--server collaborative large language model (LLM) inference, but its decoding speedup is often limited by the vocabulary-scale communication overhead. In existing distributed speculative decoding methods, exact rejection sampling requires transmitting full token distributions over the LLM vocabulary between the device and the server, which becomes costly in bandwidth-constrained environments. To address this issue, we propose DISCO, a communication-efficient framework for distributed speculative decoding, that eliminates vocabulary-scale distribution transfer while preserving the exact target distribution. DISCO achieves this by introducing a server-side replica of the draft small language model (SLM), which enables the server to locally obtain the draft-side distribution required for rejection sampling. As a result, DISCO exchanges only drafted token indices and scalar probability values between the device and the server. Experimental results on multiple datasets show that DISCO consistently achieves higher decoding speedup than existing distributed speculative decoding methods, particularly under stringent bandwidth constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑