arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.15766cs.LGcs.CL
收藏

学习调和表示用于投机采样

Learning Harmonized Representations for Speculative Sampling

Xiaohongshu Inc.(小红书公司)

Lefan Zhang, Xiaodan Wang, Yanhua Huang, Ruiwen Xu

AI总结:

针对投机采样中训练与解码上下文及目标不一致的问题,提出HASS方法,通过调和表示学习加速LLM解码,在四个LLaMA模型上实现2.81x-4.05x加速,优于EAGLE-2。

摘要:

投机采样是加速大型语言模型(LLMs)解码阶段的一种有前景的方法。近期利用目标LLM上下文信息(如隐藏状态和KV缓存)的进展已显示出显著的实践改进。然而,这些方法在训练和解码之间存在上下文不一致的问题。我们还观察到现有投机采样方法中训练目标与解码目标之间的另一种差异。在本工作中,我们提出了一种名为调和投机采样(HASS)的解决方案,通过学习调和表示来解决这些问题。HASS通过调和目标蒸馏和调和上下文对齐来加速解码阶段,且不增加推理开销。在四个LLaMA模型上的实验表明,HASS在三个数据集上平均实现了2.81倍至4.05倍的墙钟时间加速比,超过了EAGLE-2达8%至20%。代码可在https://github.com/HArmonizedSS/HASS获取。

英文摘要:

Speculative sampling is a promising approach to accelerate the decoding stage for Large Language Models (LLMs). Recent advancements that leverage target LLM's contextual information, such as hidden states and KV cache, have shown significant practical improvements. However, these approaches suffer from inconsistent context between training and decoding. We also observe another discrepancy between the training and decoding objectives in existing speculative sampling methods. In this work, we propose a solution named HArmonized Speculative Sampling (HASS) that learns harmonized representations to address these issues. HASS accelerates the decoding stage without adding inference overhead through harmonized objective distillation and harmonized context alignment. Experiments on four LLaMA models demonstrate that HASS achieves 2.81x-4.05x wall-clock time speedup ratio averaging across three datasets, surpassing EAGLE-2 by 8%-20%. The code is available at https://github.com/HArmonizedSS/HASS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑