面向低资源场景的高性价比注意力机制：线性变换的必要性与充分性
Cost-Effective Attention Mechanisms for Low Resource Settings: Necessity & Sufficiency of Linear Transformations
AI总结：
针对 SDPA 在低资源场景下计算与内存开销过大的问题，提出通过删减或新增线性变换构建三种轻量注意力变体，在保持通用性的同时显著减少参数，并使 Super Attention 在更快、更轻的情况下最高提升 10% 性能。
摘要：
从自然语言处理到视觉，缩放点积注意力（SDPA）已成为大多数现代深度学习应用的骨干。遗憾的是，其内存与计算需求在低资源场景下可能令人难以承受。本文在不牺牲其通用性的前提下提升其效率。我们提出三种注意力变体，分别移除连续的线性变换或加入一种新颖的线性变换，并在一系列标准 NLP 与视觉任务上评估它们。所提模型显著轻于标准 SDPA（参数量减少 25–50%）。结果表明，相对于模型规模的缩减，这些改变带来的性能代价可以忽略；其中一种变体（Super Attention）甚至成功实现最高 10% 的性能超越，同时提升速度并将参数减少 25%。
英文摘要：
From natural language processing to vision, Scaled Dot Product Attention (SDPA) is the backbone of most modern deep learning applications. Unfortunately, its memory and computational requirements can be prohibitive in low-resource settings. In this paper, we improve its efficiency without sacrificing its versatility. We propose three attention variants where we remove consecutive linear transformations or add a novel one, and evaluate them on a range of standard NLP and vision tasks. Our proposed models are substantially lighter than standard SDPA (and have 25-50% fewer parameters). We show that the performance cost of these changes is negligible relative to size reduction and that in one case (Super Attention) we succeed in outperforming SDPA by up to 10% while improving its speed and reducing its parameters by 25%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
