拉普拉斯头通过平滑标记表示来改进变换器
Laplacian Heads Improve Transformers by Smoothing Token Representations
AI总结:
本文提出在变换器中引入拉普拉斯头,通过平滑标记表示提升性能,发现特定形式的平滑对任务有帮助。
摘要:
变换器通过多头注意力和残差连接更新标记表示,公式为$X \leftarrow X + \sum_{i} P^{(i)}XW_{V_i}W_{o_i}$,其中$P^{(i)}$是第i个头的softmax注意力矩阵。我们提出用拉普拉斯$I - P^{(i)}$替代部分$P^{(i)}$,公式变为$X \leftarrow X + \sum_{i \in \mathcal{A}} P^{(i)}XW_{V_i}W_{o_i} + \sum_{i \in \mathcal{L}} (I - P^{(i)})XW_{V_i}W_{o_i}$。我们的方法有两个动机:首先,注意力头可更新标记表示的均值,而拉普拉斯头可直接控制序列内方差;其次,若将标记视为图中的节点,边权重为$P^{(i)}$,则$I - P^{(i)}$是图拉普拉斯矩阵,更新可解释为图上的热扩散一步。我们证明这种简单修改在监督学习、语言模型和自监督学习任务中均提升了性能。为探究原因,我们分析了有无拉普拉斯头的标记表示。在监督学习中,拉普拉斯头使同序列标记表示坍缩,并将序列均值对齐于神经崩溃几何。在语言模型中,它们增加共享相同下一个标记预测的标记表示分离度。在自监督学习中,它们产生主成分更适合分割的标记表示。在不同模态中,它们也导致更快衰减的谱,表明更强的标记平滑。总体而言,我们的发现挑战了传统观点,即标记过度平滑本质上有害,显示某些形式的平滑反而有益。
英文摘要:
Transformers update token representations through multi-head attention and residual connections: $X \leftarrow X + \sum_{i} P^{(i)}\text{Norm}(X)W_{V_i}W_{o_i}$, where $P^{(i)}$ is the softmax probability matrix in head $i$. We propose replacing some $P^{(i)}$'s with $I - P^{(i)}$, giving $X \leftarrow X + \sum_{i \in \mathcal{A}} P^{(i)}\text{Norm}(X)W_{V_i}W_{o_i} + \sum_{i \in \mathcal{L}} (I - P^{(i)})\text{Norm}(X)W_{V_i}W_{o_i}$. We call heads that use $I - P^{(i)}$ Laplacian heads. Our proposal has two motivations. First, it allows attention heads to update the mean of token representations, while Laplacian heads can directly control within-sequence variance. Second, if tokens are viewed as nodes in a graph with normalized edge weights $P^{(i)}$, then $I - P^{(i)}$ is a graph Laplacian operator, and the update can be interpreted as one step of heat diffusion on the graph. We show that this simple modification improves performance across supervised learning, causal language modeling, and vision self-supervised learning tasks. To investigate why, we examine the token representations learned with and without Laplacian heads. In supervised learning, Laplacian heads collapse token representations within the same sequence and align the sequence means with the geometry of Neural Collapse. In language modeling, they increase the separability of token representations that share the same next-token prediction. In vision self-supervised learning, they produce token representations whose top principal components better capture segmentation structure. Across modalities, they also lead to faster-decaying spectra, indicating stronger token smoothing. Overall, our findings challenge the prevailing view that token oversmoothing is inherently harmful, showing instead that certain forms of smoothing can be beneficial.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。