LongQLoRA:一种高效扩展大语言模型上下文长度的有效方法
LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models
AI总结:
本研究提出高效的LongQLoRA方法,融合位置插值、QLoRA与移位短注意力技术,以极低训练资源扩展大模型上下文长度,在多个数据集上取得优异表现,同时构建长指令数据优化Vicuna模型并开源相关资源。
摘要:
我们提出了LongQLoRA,这是一种高效且有效的大语言模型上下文长度扩展方法,所需训练资源更少。LongQLoRA融合了位置插值(Position Interpolation)、QLoRA以及LongLoRA的移位短注意力(Shift Short Attention)的优势。仅需单张32GB V100 GPU,LongQLoRA就能在1000个微调步内将LLaMA2 7B和13B的上下文长度从4096扩展至8192,甚至可达12k。在PG19和Proof-pile数据集上,LongQLoRA取得了具有竞争力的困惑度表现:在8192的评估上下文长度下,我们的模型优于LongLoRA,且表现非常接近MPT-7B-8K。我们收集并构建了39k条长指令数据,将Vicuna-13B的上下文长度从4096扩展至8192,在长上下文和短上下文生成任务中均取得了良好表现。我们还开展了消融实验,研究LoRA秩、微调步数以及推理时注意力模式的影响。模型权重、训练数据和代码可在https://github.com/yangjianxin1/LongQLoRA 获取。
英文摘要:
We present LongQLoRA, an efficient and effective method to extend context length of large language models with less training resources. LongQLoRA combines the advantages of Position Interpolation, QLoRA and Shift Short Attention of LongLoRA. With a single 32GB V100 GPU, LongQLoRA can extend the context length of LLaMA2 7B and 13B from 4096 to 8192 and even to 12k within 1000 finetuning steps. LongQLoRA achieves competitive perplexity performance on PG19 and Proof-pile datasets, our model outperforms LongLoRA and is very close to MPT-7B-8K within the evaluation context length of 8192. We collect and build 39k long instruction data to extend context length of Vicuna-13B from 4096 to 8192 and achieve good performance both in long and short context generation task. We also do some ablation experiments to study the effect of LoRA rank, finetuning steps and attention patterns in inference.The model weights, training data and code are avaliable at https://github.com/yangjianxin1/LongQLoRA.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。