arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.08618cs.CL
收藏

Zebra:通过分层分组局部-全局注意力扩展上下文窗口

Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention

Tencent AI Lab(腾讯人工智能实验室)

Kaiqiang Song, Xiaoyang Wang, Sangwoo Cho, Xiaoman Pan, Dong Yu

AI总结:

针对大语言模型扩展上下文窗口的难题,提出Zebra架构,通过分层分组局部-全局注意力平衡注意力分配,降低计算与内存开销,在长短序列任务上性能优异且效率更高。

摘要:

本文提出了一种提升大语言模型(LLM)处理与理解长文本序列能力的新方法,这一能力是需要深度理解和整合大量信息的应用场景的关键要素。针对以Transformer架构为核心的大语言模型在扩展上下文窗口时存在的固有挑战,我们提出了名为Zebra的新型模型架构。该架构采用分组局部-全局注意力层,有效解决了Transformer全注意力机制带来的时间与内存复杂度呈二次方增长的问题。我们的模型类似斑马的交替条纹,平衡了局部与全局注意力层,显著降低了计算需求与内存消耗。我们开展了全面实验评估Zebra的性能,包括从零预训练、长上下文适配训练续训以及长指令微调。结果表明,Zebra在短序列与长序列基准测试上均取得了相当或更优的性能,同时还提升了训练与推理效率。

英文摘要:

This paper introduces a novel approach to enhance the capabilities of Large Language Models (LLMs) in processing and understanding extensive text sequences, a critical aspect in applications requiring deep comprehension and synthesis of large volumes of information. Recognizing the inherent challenges in extending the context window for LLMs, primarily built on Transformer architecture, we propose a new model architecture, referred to as Zebra. This architecture efficiently manages the quadratic time and memory complexity issues associated with full attention in the Transformer by employing grouped local-global attention layers. Our model, akin to a zebra's alternating stripes, balances local and global attention layers, significantly reducing computational requirements and memory consumption. Comprehensive experiments, including pretraining from scratch, continuation of long context adaptation training, and long instruction tuning, are conducted to evaluate the Zebra's performance. The results show that Zebra achieves comparable or superior performance on both short and long sequence benchmarks, while also enhancing training and inference efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑