arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.03493cs.LGcs.CL
收藏

LASER:具有指数变换的注意力机制

LASER: Attention with Exponential Transformation

University of Texas at Austin(德克萨斯大学奥斯汀分校);Google(谷歌)

Sai Surya Duvvuri, Inderjit S. Dhillon

AI总结:

针对Transformer中softmax注意力梯度信号微弱导致学习低效的问题,提出LASER注意力机制以增大梯度信号,在大型语言模型及视觉、语音等多模态任务上显著提升了性能与泛化能力。

摘要:

Transformer在多个序列相关任务中产生了巨大影响,这在很大程度上归功于其能够通过基于softmax的点积注意力从序列的任意部分进行检索。该机制在Transformer的性能中发挥了至关重要的作用。我们分析了在注意力机制中通过softmax操作反向传播的梯度,并观察到这些梯度通常很小。这种不良的梯度信号反向传播会导致对注意力操作之前的参数进行低效的学习。为此,我们引入了一种名为LASER的新注意力机制,并通过解析证明其能提供更大的梯度信号。我们表明,通过对现有的注意力实现进行微小修改即可实现LASER注意力。我们在参数量高达77亿的自回归大型语言模型(LLM)上进行了实验,在下游评估中较标准注意力平均提升了1.44%,微调提升了1.65%。此外,LASER在各种任务(视觉、文本和语音)中展现出泛化性能的提升:在ImageNet上的Vision Transformer (ViT)、在Librispeech语音转文本上的Conformer,以及具有22亿参数的BERT。

英文摘要:

Transformers have had tremendous impact for several sequence related tasks, largely due to their ability to retrieve from any part of the sequence via softmax based dot-product attention. This mechanism plays a crucial role in Transformer's performance. We analyze the gradients backpropagated through the softmax operation in the attention mechanism and observe that these gradients can often be small. This poor gradient signal backpropagation can lead to inefficient learning of parameters preceeding the attention operations. To this end, we introduce a new attention mechanism called LASER, which we analytically show to admit a larger gradient signal. We show that LASER attention can be implemented by making small modifications to existing attention implementations. We conduct experiments on autoregressive large language models (LLMs) with upto 7.7 billion parameters with an average improvement of upto 1.44% over standard attention on downstream evaluations and 1.65% finetuning improvements. Additionally, LASER demonstrates generalization performance improvement across a variety of tasks (vision, text and speech):Vision Transformer (ViT) on Imagenet, Conformer on the Librispeech speech-to-text and BERT with 2.2 billion parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑