arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.07004cs.CL
收藏

用100个样本扩展大语言模型的上下文窗口

Extending LLMs' Context Window with 100 Samples

Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Lab(上海人工智能实验室);Generative AI Research Lab (GAIR)(生成式人工智能研究实验室(GAIR))

Yikai Zhang, Junlong Li, Pengfei Liu

AI总结:

本文提出一种结合调整RoPE基频与缩放注意力logits的方法,通过保持注意力熵稳定,仅用100个样本和6步训练便将LLaMA-2-7B-Chat上下文窗口扩展至16,384,并验证了其高效性与鲁棒性。

摘要:

大语言模型(LLMs)在超出其预训练上下文窗口后外推能力有限,这限制了它们在长输入下游任务中的应用。近期研究试图通过修改旋转位置嵌入(RoPE)来扩展LLMs的上下文窗口,RoPE是LLaMA、PaLM和GPT-NeoX等知名LLMs采用的主流位置编码方法。然而,先前的工作如Position Interpolation(PI)和YaRN资源消耗大,且缺乏比较实验来评估其适用性。在本工作中,我们识别出LLMs的注意力熵(即注意力分数的信息熵)需要保持稳定的内在需求,并提出一种新的RoPE扩展方法,该方法结合调整RoPE的基频和缩放注意力logits,帮助LLMs高效适应更大的上下文窗口。我们在各种上下文需求任务上验证了我们的方法在不同上下文窗口大小下的微调性能和鲁棒性方面的优越性。值得注意的是,我们的方法仅用100个样本和6个训练步骤就将LLaMA-2-7B-Chat的上下文窗口扩展到16,384,展示了非凡的效率。最后,我们还探讨了数据组成和训练课程如何影响特定下游任务的上下文窗口扩展,建议以长对话微调LLMs作为一个良好的起点。我们在https://github.com/GAIR-NLP/Entropy-ABF发布了代码和SFT数据。

英文摘要:

Large Language Models (LLMs) are known to have limited extrapolation ability beyond their pre-trained context window, constraining their application in downstream tasks with lengthy inputs. Recent studies have sought to extend LLMs' context window by modifying rotary position embedding (RoPE), a popular position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are resource-intensive and lack comparative experiments to assess their applicability. In this work, we identify the inherent need for LLMs' attention entropy (i.e. the information entropy of attention scores) to maintain stability and introduce a novel extension to RoPE which combines adjusting RoPE's base frequency and scaling the attention logits to help LLMs efficiently adapt to a larger context window. We validate the superiority of our method in both fine-tuning performance and robustness across different context window sizes on various context-demanding tasks. Notably, our method extends the context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6 training steps, showcasing extraordinary efficiency. Finally, we also explore how data compositions and training curricula affect context window extension for specific downstream tasks, suggesting fine-tuning LLMs with lengthy conversations as a good starting point. We release our code and SFT data at https://github.com/GAIR-NLP/Entropy-ABF.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑