改变乘积，保持参数：Transformer的关联代数层
Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers
浏览论文内容
中文总结 AI 辅助
本文提出用关联代数层替换Transformer中的稠密矩阵乘法，在保持参数的同时降低计算复杂度，实验显示吞吐量提升6.2%-7.8%，但下游指标略降，验证了该方法的可行性与可训练性。
中文摘要 AI 辅助
快速矩阵乘法算法保持乘积不变，并寻找更廉价的评估方式。我们转而探讨Transformer学得的投影是否可以使用另一种更廉价的乘积。基于一种关联代数构造，该构造用相同权重块上的更稀疏交互表替换普通矩阵乘法，我们构建了一个在物理块大小固定时矩阵维度上具有二次算术复杂度的家族，并推导了适用于GPU执行的有限形状约束。该构造通过Alder-Strassen界证明了其双线性秩的最优性，并可实现为与因果掩码和KV缓存解码兼容的行类型矩形投影。我们通过训练两个约1.1亿参数、仅解码器的Transformer语言模型（使用相同配方和123亿token预算，仅在前馈层上有所不同：一个使用普通稠密矩阵乘法，另一个使用关联代数乘积）对该方法进行了实证测试。在四个提示域中，代数模型实现了6.2%至7.8%的端到端生成吞吐量提升，同时在所有三个报告的下游指标上获得较低分数。我们将这些结果视为所提方法在小规模下的可行性和可训练性检查，将进一步研究留待未来工作。
英文摘要
Fast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether. Building on an associative-algebra construction that replaces ordinary matrix multiplication with a sparser interaction table over the same weight blocks, we construct a family with quadratic arithmetic in the matrix dimension when the physical block size remains fixed, and derive finite-shape constraints for GPU execution. The construction is provably optimal for its bilinear rank by the Alder--Strassen bound and can be realized as row-typed rectangular projections compatible with causal masking and KV-cached decoding. We provide an empirical test of this approach by training two approximately 110M-parameter decoder-only Transformer LMs from the same recipe and 12.3B-token budget, differing only in their feed-forward layer: one uses ordinary dense matrix multiplication and the other uses the associative-algebra product. Across four prompt domains, the algebraic model achieves a 6.2--7.8\% increase in end-to-end generation throughput, while obtaining lower scores on all three reported downstream metrics. We treat these results as a feasibility and trainability check for the proposed approach at small scale, leaving further investigation to future work.
