通过利用对称性的泰勒近似实现常数成本的自注意力机制
Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
AI总结:
通过利用对称性的泰勒近似,实现自注意力机制的常数成本计算,显著降低大规模Transformer模型的基础设施和能源需求。
摘要:
当今最广泛使用的AI模型是采用自注意力机制的Transformer。在标准形式下,自注意力机制的成本随着上下文长度的增加而增加,推动了对存储、计算和能源需求的增长,这些需求如今已超过了社会所能提供的能力。为了帮助解决这一问题,我们证明自注意力机制可以以恒定的每token成本高效计算,达到数量级的内存使用和计算减少。我们通过将传统形式的泰勒展开分解为对称张量积链的表达式来推导我们的公式。我们利用其对称性,获得高效的前馈转换,将查询和键高效地映射到最小多项式核特征基的坐标中。值得注意的是,成本与头大小成反比,这使得每个token可以应用更多的头数。我们实现了我们的公式并经验验证了其正确性。我们的工作使在适度固定成本下实现无限制的token生成成为可能,大大减少了大规模Transformer模型的基础设施和能源需求。我们引入的数学技术具有独立的研究价值。
英文摘要:
The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attention incurs costs that increase with context length, driving demand for storage, compute, and energy that is now outstripping society's ability to provide them. To help address this issue, we show that self-attention is efficiently computable to arbitrary precision with constant cost per token, achieving orders-of-magnitude reductions in memory use and computation. We derive our formulation by decomposing the conventional formulation's Taylor expansion into expressions over symmetric chains of tensor products. We exploit their symmetry to obtain feed-forward transformations that efficiently map queries and keys to coordinates in a minimal polynomial-kernel feature basis. Notably, cost is fixed inversely in proportion to head size, enabling application over a greater number of heads per token than otherwise feasible. We implement our formulation and empirically validate its correctness. Our work enables unbounded token generation at modest fixed cost, substantially reducing the infrastructure and energy demands of large-scale Transformer models. The mathematical techniques we introduce are of independent interest.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。