注意力的夸克

摘要：

注意力在自然和人工智能系统中都起着基础性的作用。在深度学习中，基于注意力的神经架构，如Transformer架构，被广泛用于解决自然语言处理及其他领域的问题。在这里，我们调查的基本积木的注意力和它们的计算特性。在深度学习的标准模型中，我们根据注意力的来源、目标和计算机制对所有可能的基本构建块进行了分类。我们确定并研究了三种最重要的机制：加法激活注意，乘法输出注意（输出门控）和乘法突触注意（突触门控）。门控机制对应于标准模型的乘法扩展，并在所有当前基于注意力的深度学习架构中使用。我们研究了它们的功能特性，并估计了线性和多项式阈值门的情况下几个注意力构建块的能力。令人惊讶的是，添加剂激活注意力在下界的证明中起着核心作用。注意力机制减少了某些基本回路的深度，并在不产生全部成本的情况下利用了二次激活的力量。

英文摘要：

Attention plays a fundamental role in both natural and artificial intelligence systems. In deep learning, attention-based neural architectures, such as transformer architectures, are widely used to tackle problems in natural language processing and beyond. Here we investigate the fundamental building blocks of attention and their computational properties. Within the standard model of deep learning, we classify all possible fundamental building blocks of attention in terms of their source, target, and computational mechanism. We identify and study three most important mechanisms: additive activation attention, multiplicative output attention (output gating), and multiplicative synaptic attention (synaptic gating). The gating mechanisms correspond to multiplicative extensions of the standard model and are used across all current attention-based deep learning architectures. We study their functional properties and estimate the capacity of several attentional building blocks in the case of linear and polynomial threshold gates. Surprisingly, additive activation attention plays a central role in the proofs of the lower bounds. Attention mechanisms reduce the depth of certain basic circuits and leverage the power of quadratic activations without incurring their full cost.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。