重新思考密集线性变换：用于神经网络近线性训练的分阶段成对混合（SPM）
Rethinking Dense Linear Transformations: Stagewise Pairwise Mixing (SPM) for Near-Linear Training in Neural Networks
AI总结：
本文提出SPM方法，通过分阶段成对混合替代密集线性层，实现近线性训练，提升模型泛化能力并减少计算成本。
摘要：
密集线性层是现代机器学习模型中计算和参数成本的主要来源，尽管其二次复杂度往往与所学表示的组合结构不匹配。我们引入了分阶段成对混合（SPM），这是一种结构化的线性运算，用稀疏成对混合阶段代替密集矩阵。一个SPM层在O(nL)时间内实现全局线性变换，具有O(nL)个参数，其中L通常是常数或log₂n。SPM允许精确的闭合形式前向和反向计算。SPM被设计为可以无缝替换前馈网络、递归架构、注意力机制等中的密集线性层。我们为两种参数化推导了完整的前向和反向表达式：一种是正交规范保持的旋转变体，另一种是完全通用的2×2混合变体。除了计算节省外，SPM的分阶段结构诱导了显式的组合归纳偏差，当与任务结构对齐时，它会限制模型容量并提高泛化能力。我们展示了概念验证实验，证明了在结构化学习问题中显著减少了实时时钟成本并提高了准确性，同时在现实世界基准测试中保持了有竞争力的性能。
英文摘要：
Dense linear layers are a dominant source of computational and parametric cost in modern machine learning models, despite their quadratic complexity and often being misaligned with the compositional structure of learned representations. We introduce Stagewise Pairwise Mixers (SPM), a structured linear operator that replaces dense matrices with a composition of sparse pairwise-mixing stages. An SPM layer implements a global linear transformation in $O(nL)$ time with $O(nL)$ parameters, where $L$ is typically constant or $log_2n$, and admits exact closed-form forward and backward computations. SPM is designed as a drop-in replacement for dense linear layers in feedforward networks, recurrent architectures, attention mechanisms, etc. We derive complete forward and backward expressions for two parameterizations: an orthogonal norm-preserving rotation-based variant and a fully general $2 \times 2$ mixing variant. Beyond computational savings, the stagewise structure of SPM induces an explicit compositional inductive bias that constrains model capacity and improves generalization when aligned with task structure. We present proof-of-concept experiments demonstrating substantial reductions in wall-clock cost and improved accuracy on structured learning problems, while retaining competitive performance on real-world benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
