仿射缩放注意力：迈向灵活且稳定的Transformer注意力
Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
AI总结：
仿射缩放注意力通过引入输入依赖的缩放和偏置项，提升Transformer模型的训练稳定性与注意力行为控制能力。
摘要：
Transformer注意力通常通过softmax归一化实现，这强制注意力权重具有单位和归一化。虽然在许多情况下有效，但这一约束可能限制在控制注意力大小时的灵活性，并可能在训练过程中导致过于集中或不稳定的注意力模式。先前的工作探索了诸如注意力沉底或门控机制的修改，但这些方法只提供有限或间接的注意力重新加权控制。我们提出仿射缩放注意力，这是一种标准注意力的简单扩展，引入了输入依赖的缩放和相应的偏置项，应用于softmax归一化的注意力权重。这种设计放松了严格的归一化约束，同时保持了价值表示的聚合，允许模型在受控的情况下调整注意力的相对分布和尺度。我们通过大规模语言模型预训练在多个模型大小上经验性地评估了仿射缩放注意力。实验结果表明，与标准softmax注意力和注意力沉底基线相比，训练稳定性、优化行为和下游任务性能都有持续的改进。这些发现表明，对注意力输出的适度重新加权提供了一种实用且有效的方法来改进Transformer模型中的注意力行为。
英文摘要：
Transformer attention is typically implemented using softmax normalization, which enforces attention weights with unit sum normalization. While effective in many settings, this constraint can limit flexibility in controlling attention magnitudes and may contribute to overly concentrated or unstable attention patterns during training. Prior work has explored modifications such as attention sinks or gating mechanisms, but these approaches provide only limited or indirect control over attention reweighting. We propose Affine-Scaled Attention, a simple extension to standard attention that introduces input-dependent scaling and a corresponding bias term applied to softmax-normalized attention weights. This design relaxes the strict normalization constraint while maintaining aggregation of value representations, allowing the model to adjust both the relative distribution and the scale of attention in a controlled manner. We empirically evaluate Affine-Scaled Attention in large-scale language model pretraining across multiple model sizes. Experimental results show consistent improvements in training stability, optimization behavior, and downstream task performance compared to standard softmax attention and attention sink baselines. These findings suggest that modest reweighting of attention outputs provides a practical and effective way to improve attention behavior in Transformer models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
