arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17463cs.CL
收藏

大语言模型的免训练长上下文扩展

Training-Free Long-Context Scaling of Large Language Models

The University of Hong Kong(香港大学);Alibaba Group(阿里巴巴集团);Fudan University(复旦大学)

Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, Lingpeng Kong

AI总结:

提出双块注意力(DCA),免训练扩展LLM上下文至100k+令牌,性能媲美微调模型,开源替代方案。

摘要:

当输入令牌数量超过预训练长度时,大语言模型(LLMs)处理和生成连贯文本的能力会显著减弱。鉴于对大规模模型进行更长序列的微调成本高昂,我们提出了双块注意力(Dual Chunk Attention, DCA),该方法使Llama2 70B无需持续训练即可支持超过10万令牌的上下文窗口。通过将长序列的注意力计算分解为基于块的模块,DCA能够有效捕获同一块内(块内)和不同块之间(块间)令牌的相对位置信息,并与Flash Attention无缝集成。除了其令人印象深刻的扩展能力外,DCA在实际长上下文任务上取得的性能与微调模型相当甚至更优。与专有模型相比,我们免训练的70B模型达到了gpt-3.5-16k性能的94%,表明其是一个可行的开源替代方案。本工作使用的所有代码和数据已在\url{https://github.com/HKUNLP/ChunkLlama}发布。

英文摘要:

The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length. Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training. By decomposing the attention computation for long sequences into chunk-based modules, DCA manages to effectively capture the relative positional information of tokens within the same chunk (Intra-Chunk) and across distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash Attention. In addition to its impressive extrapolation capability, DCA achieves performance on practical long-context tasks that is comparable to or even better than that of finetuned models. When compared with proprietary models, our training-free 70B model attains 94% of the performance of gpt-3.5-16k, indicating it is a viable open-source alternative. All code and data used in this work are released at \url{https://github.com/HKUNLP/ChunkLlama}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑