arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.19399cs.CLcs.AIcs.LG
收藏

Scalable-Softmax在注意力机制中更具优势

Scalable-Softmax Is Superior for Attention

Institute for Physics of Intelligence, The University of Tokyo(东京大学智能物理研究所)

Ken M. Nakanishi

AI总结:

针对Transformer中Softmax导致长上下文注意力分布扁平化的问题,提出Scalable-Softmax(SSMax),显著提升模型预训练收敛速度、长上下文性能及长度泛化能力。

摘要:

随着输入向量尺寸的增加,Softmax函数输出的向量最大元素趋近于零。基于Transformer的语言模型依赖Softmax计算注意力分数,导致上下文长度增加时注意力分布趋于扁平化。这削弱了模型有效聚焦关键信息的能力,并可能限制其长度泛化性能。为解决此问题,我们提出Scalable-Softmax(SSMax),用于替换输入向量尺寸动态变化场景下的Softmax。SSMax可无缝集成至现有基于Transformer的架构中。语言建模实验表明,使用SSMax的模型不仅在预训练阶段实现更快的损失下降,还在长上下文处理与关键信息检索任务中显著提升性能。注意力分数分析进一步揭示,SSMax使模型即使在长上下文中也能聚焦关键信息。此外,尽管预训练初期即采用SSMax的模型能获得更优的长度泛化能力,但已启动预训练的模型仍可通过在注意力层将Softmax替换为SSMax(无论在预训练中或预训练后)来部分获得此能力。

英文摘要:

The maximum element of the vector output by the Softmax function approaches zero as the input vector size increases. Transformer-based language models rely on Softmax to compute attention scores, causing the attention distribution to flatten as the context size grows. This reduces the model's ability to prioritize key information effectively and potentially limits its length generalization. To address this problem, we propose Scalable-Softmax (SSMax), which replaces Softmax in scenarios where the input vector size varies. SSMax can be seamlessly integrated into existing Transformer-based architectures. Experimental results in language modeling show that models using SSMax not only achieve faster loss reduction during pretraining but also significantly improve performance in long contexts and key information retrieval. Furthermore, an analysis of attention scores reveals that SSMax enables the model to focus attention on key information even in long contexts. Additionally, although models that use SSMax from the beginning of pretraining achieve better length generalization, those that have already started pretraining can still gain some of this ability by replacing Softmax in the attention layers with SSMax, either during or after pretraining.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑