arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.08263cs.CL
收藏

Fast Chain-of-Thought:从并行解码中对未来的一瞥带来更快的答案

Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster

Nanjing University(南京大学);Ant Group(蚂蚁集团)

Hongxuan Zhang, Zhining Liu, Yao Zhao, Jiaqi Zheng, Chenyi Zhuang, Jinjie Gu, Guihai Chen

AI总结:

提出FastCoT框架,通过大小可变上下文窗口结合并行与自回归解码,在不修改LLM的前提下节省近20%推理时间且性能基本无损。

摘要:

在这项工作中,我们提出了FastCoT,这是一个基于并行解码的模型无关框架,无需对辅助模型进行进一步训练,也无需修改LLM本身。FastCoT使用一个大小随位置变化的大小可变上下文窗口,同时进行并行解码和自回归解码,从而充分利用GPU计算资源。在FastCoT中,并行解码部分为LLM提供了由近似token组成的对未来的一瞥,与因果transformer使用的常规自回归解码相比,这可以带来更快的答案。我们还提供了LLM内并行解码的实现,支持KV-cache生成和批处理。通过大量实验,我们证明与常规方法相比,FastCoT节省了近20%的推理时间,且仅有可忽略的性能下降。此外,我们表明上下文窗口大小对不同任务表现出相当的鲁棒性。

英文摘要:

In this work, we propose FastCoT, a model-agnostic framework based on parallel decoding without any further training of an auxiliary model or modification to the LLM itself. FastCoT uses a size-varying context window whose size changes with position to conduct parallel decoding and auto-regressive decoding simultaneously, thus fully utilizing GPU computation resources. In FastCoT, the parallel decoding part provides the LLM with a quick glance of the future composed of approximate tokens, which could lead to faster answers compared to regular autoregressive decoding used by causal transformers. We also provide an implementation of parallel decoding within LLM, which supports KV-cache generation and batch processing. Through extensive experiments, we demonstrate that FastCoT saves inference time by nearly 20% with only a negligible performance drop compared to the regular approach. Additionally, we show that the context window size exhibits considerable robustness for different tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑