UIO-LLMs:长上下文大语言模型的无偏增量优化
UIO-LLMs: Unbiased Incremental Optimization for Long-Context LLMs
AI总结:
针对大语言模型长文本处理难题,提出UIO-LLMs无偏增量优化方法,通过记忆增强和TBPTT算法,将Llama2-7b-chat上下文扩展至100K,参数仅增2%,推理近线性。
摘要:
由于上下文窗口大小有限,管理长文本对大语言模型(LLMs)来说颇具挑战。本研究引入了UIO-LLMs,一种在长上下文设置下用于记忆增强型Transformer的无偏增量优化方法。我们首先将该过程概念化为一个简化的编码器-解码器框架,其中权重共享的编码器和解码器分别将上下文段封装为记忆,并利用这些记忆来预测后续段的输出。随后,通过将我们的记忆增强型Transformer视为全连接循环神经网络(RNNs),我们使用截断时间反向传播(TBPTT)算法来改进训练过程,该算法融入了创新的增量优化技术。这些技术不仅降低了时间复杂度,还通过无偏优化过程解决了梯度计算中的偏差问题。UIO-LLMs成功处理了长上下文,例如将Llama2-7b-chat的上下文窗口从4K扩展到100K个令牌,仅增加了2%的额外参数,同时随着上下文长度的增加,推理成本几乎保持线性增长。
英文摘要:
Managing long texts is challenging for large language models (LLMs) due to limited context window sizes. This study introduces UIO-LLMs, an unbiased incremental optimization approach for memory-enhanced transformers under long-context settings. We initially conceptualize the process as a streamlined encoder-decoder framework where the weights-shared encoder and decoder respectively encapsulate a context segment into memories and leverage these memories to predict outputs of the subsequent segment. Subsequently, by treating our memory-enhanced transformers as fully-connected recurrent neural networks (RNNs), we refine the training process using the Truncated Backpropagation Through Time (TBPTT) algorithm, which incorporates innovative incremental optimization techniques. These techniques not only diminish time complexity but also address the bias in gradient computation through an unbiased optimization process. UIO-LLMs successfully handle long context, such as extending the context window of Llama2-7b-chat from 4K to 100K tokens with minimal 2% additional parameters, while keeping the inference cost nearly linear as context length increases.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。