arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.16758cs.CL
收藏

迈向快速多语言LLM推理:投机解码与专用起草模型

Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters

KAIST AI(韩国科学技术院人工智能学院)

Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun

AI总结:

本文针对多语言LLM推理慢的问题,提出在投机解码中采用预训练和微调策略优化的语言专用起草模型,显著提升了多语言推理速度,并通过推理时间、域外加速及GPT-4o评估验证了其有效性。

摘要:

大型语言模型(LLMs)彻底改变了自然语言处理,并扩展了其在各种商业应用中的适用性。然而,这些模型的部署在多语言环境中受到高推理时间的限制。为了缓解这一挑战,本文探索了投机解码中助手模型的训练方案,该方案用于起草,然后其未来token由目标LLM验证。我们表明,通过有针对性的预训练和微调策略优化的语言特定起草模型,与以前的方法相比,显著加快了推理时间。我们在各种语言中验证了这些模型在推理时间、域外加速和GPT-4o评估方面的表现。

英文摘要:

Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑