ETT:在测试时扩展大语言模型的长上下文理解能力
ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time
AI总结:
针对Transformer模型处理长序列时二次方开销的问题,本文提出ETT方法,通过在测试时对分块输入高效微调模型参数,将上下文长度扩展32倍,准确率提升30%,并发现微调FFN第二层更有效。
摘要:
基于Transformer的语言模型的计算和内存开销随序列长度呈二次方增长。这种二次方成本在处理长序列时给大语言模型(LLM)的应用带来了挑战。在这项工作中,我们引入了\ourmodelacronym(在测试时扩展),一种用于扩展短上下文基于Transformer的LLM上下文长度的方法,其内存需求恒定,计算开销为线性。ETT通过在输入上下文上高效微调模型参数(将输入上下文分块为重叠的小子序列),在测试时实现上下文长度的扩展。我们在LongBench上评估了ETT,将GPT-Large和Phi-2的上下文长度从1k扩展到32k个token,即扩展了32倍。这使模型准确率最多提升了30%。我们还研究了如何有效且高效地将上下文存储在LLM的权重中。通过详细的消融研究,我们考察了哪些Transformer模块在测试时微调最为有益。有趣的是,我们发现微调FFN的第二层比全量微调更有效,进一步提升了模型的准确率。
英文摘要:
Transformer-based Language Models' computation and memory overhead increase quadratically as a function of sequence length. The quadratic cost poses challenges when employing LLMs for processing long sequences. In this work, we introduce \ourmodelacronym~(Extend at Test-Time), method for extending the context length of short context Transformer-based LLMs, with constant memory requirement and linear computation overhead. ETT enable the extension of the context length at test-time by efficient fine-tuning the model's parameters on the input context, chunked into overlapping small subsequences. We evaluate ETT on LongBench by extending the context length of GPT-Large and Phi-2 up to 32 times, increasing from 1k to 32k tokens. This results in up to a 30 percent improvement in the model's accuracy. We also study how context can be stored in LLM's weights effectively and efficiently. Through a detailed ablation study, we examine which Transformer modules are most beneficial to fine-tune at test-time. Interestingly, we find that fine-tuning the second layer of the FFNs is more effective than full fine-tuning, leading to a further improvement in the models' accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。