arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.12465cs.CLcs.LG
收藏

核心上下文感知Transformer用于长上下文语言建模

Core Context Aware Transformers for Long Context Language Modeling

South China University of Technology(华南理工大学);Peng Cheng Laboratory(鹏城实验室);Pazhou Laboratory(琶洲实验室);Harbin Institute of Technology(哈尔滨工业大学);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education(教育部大数据与智能机器人重点实验室)

Yaofo Chen, Zeng You, Shuhai Zhang, Haokun Li, Yirui Li, Yaowei Wang, Mingkui Tan

AI总结:

针对长上下文中冗余信息损害性能与效率的问题,提出即插即用的核心上下文感知注意力,通过全局池化压缩核心与局部保持模块,以极小微调成本替换自注意力,实现高效长上下文建模。

摘要:

基于Transformer的大语言模型(LLMs)在广泛任务中取得了显著成功,这主要归功于自注意力机制,该机制要求每个词元将所有前置词元视为其上下文来计算注意力。然而,当上下文长度L变得非常大(例如128K)时,上下文中潜在冗余信息的数量趋于增加。冗余上下文不仅妨碍了建模表示性能,还招致了不必要的计算和存储开销。在本文中,我们提出了一种即插即用的核心上下文感知(CCA)注意力机制,用于高效的长上下文建模,该机制包含两个互补模块:1)全局感知池化模块对输入词元进行分组,并根据其重要性将每组动态压缩为一个核心词元。通过这种方式,我们的方法在学习过程中自动聚焦并强化核心上下文,同时减少冗余,从而实现有效的长期依赖建模。2)局部保持模块融合相邻词元以保留局部上下文,用于精细表示。值得注意的是,我们的CCA注意力机制能够以极小的微调成本替换现有大语言模型中的自注意力模块。大量实验结果表明,我们的方法在长上下文建模和计算效率方面均优于最先进的方法。

英文摘要:

Transformer-based Large Language Models (LLMs) have exhibited remarkable success in extensive tasks primarily attributed to self-attention mechanism, which requires a token to consider all preceding tokens as its context to compute attention. However, when the context length L becomes very large (e.g., 128K), the amount of potentially redundant information in the context tends to increase. The redundant context not only hampers the modeling representation performance but also incurs unnecessary computational and storage overhead. In this paper, we propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-context modeling, comprising two complementary modules: 1) Globality-aware pooling module groups input tokens and dynamically compresses each group into one core token based on their significance. In this way, our method automatically focuses and strengthens core context while diminishing redundancy during the learning process, leading to effective long-term dependency modeling. 2) Locality-preserving module incorporates neighboring tokens to preserve local context for detailed representation. Notably, our CCA-Attention is able to replace the self-attention module in existing LLMs with minimal fine-tuning cost. Extensive experimental results show the superiority of our method in both long-context modeling and computational efficiency over state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑