从一叠起:多尺度自我注入用于上下文窗口扩展
Stacked from One: Multi-Scale Self-Injection for Context Window Extension
AI总结:
本文提出多尺度自我注入框架,通过压缩和查询感知信息获取扩展上下文窗口,实现高效且准确的长上下文处理。
摘要:
当代大语言模型(LLMs)的有限上下文窗口仍然是其在多样化领域广泛应用的主要瓶颈。尽管持续预训练长上下文数据提供了一种简单解决方案,但其带来高昂的数据获取和计算成本。为解决这一挑战,我们提出~\modelname,一种基于多粒度上下文压缩和查询感知信息获取的新型框架。SharedLLM包含两个堆叠的短上下文LLM:下层模型作为压缩器,上层模型作为解码器。下层模型将长输入压缩成紧凑的多粒度表示,然后转发给上层模型进行上下文感知处理。为最大化效率,此信息传输仅在最低层发生,避免了漫长的前向传递和冗余的交叉注意力操作。整个过程,其中上层和下层模型源自同一基础LLM层,称为~\textit{self-injection}。为了支持此架构,一种专用的树状数据结构实现了高效编码和查询感知的上下文信息检索。尽管仅在8K token序列上训练,~\modelname能够有效泛化到超过128K token的输入。在全面的长上下文建模和理解基准测试中,~\modelname在性能上优于或可与强基线相媲美,实现了效率和准确性的最佳平衡。此外,这些设计选择使~\modelname显著减少了内存占用,并带来了显著的推理加速(比流式处理快2倍,比编码器-解码器架构快3倍)
英文摘要:
The limited context window of contemporary large language models (LLMs) remains a primary bottleneck for their broader application across diverse domains. Although continual pre-training on long-context data offers a straightforward solution, it incurs prohibitive data acquisition and computational costs. To address this challenge, we propose~\modelname, a novel framework based on multi-grained context compression and query-aware information acquisition. SharedLLM comprises two stacked short-context LLMs: a lower model serving as a compressor and an upper model acting as a decoder. The lower model compresses long inputs into compact, multi-grained representations, which are then forwarded to the upper model for context-aware processing. To maximize efficiency, this information transfer occurs exclusively at the lowest layers, bypassing lengthy forward passes and redundant cross-attention operations. This entire process, wherein the upper and lower models are derived from the same underlying LLM layers, is termed~\textit{self-injection}. To support this architecture, a specialized tree-based data structure enables the efficient encoding and query-aware retrieval of contextual information. Despite being trained on sequences of only 8K tokens, \modelname~effectively generalizes to inputs exceeding 128K tokens. Across a comprehensive suite of long-context modeling and understanding benchmarks, \modelname~achieves performance superior or comparable to strong baselines, striking an optimal balance between efficiency and accuracy. Furthermore, these design choices allow \modelname~to substantially reduce the memory footprint and yield notable inference speedups ($2\times$ over streaming and $3\times$ over encoder-decoder architectures).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。