发表机构
Zhejiang University; Alibaba Group（浙江大学; 阿里巴巴集团）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对推荐场景中Transformer性能瓶颈问题，提出SpecFormer，通过引入可学习频谱软化模块、频谱软化注意力机制和频谱残差位置编码，有效减轻嵌入和注意力崩溃，实验证明其显著优于基线且有卓越扩展能力。
AI 中文摘要
变压器架构在各个领域都取得了显著成功，但直接将其标准自注意力机制应用于推荐往往性能欠佳。本文揭示这种性能瓶颈源于推荐场景特有的严重嵌入和注意力崩溃。推荐数据的异质性和长尾性质导致由少数主奇异值主导的严重频谱崩溃。理论证明这在推荐模型的前向和反向传播中引发恶性循环，加速嵌入和注意力崩溃并限制模型的扩展能力。为解决这些问题，提出SpecFormer，它引入可学习频谱软化模块动态平滑输入令牌嵌入的奇异值分布，频谱软化注意力机制在更均匀频谱分布空间中建模特征交互，以及通过奇异值泰勒展开的频谱残差位置编码为特征交互提供频谱归纳偏差。在一个工业和两个公共数据集上的大量实验表明SpecFormer显著优于现有基线，且已成功部署在实际商业推荐系统中并展现出卓越扩展能力。
英文摘要
Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.
Comments12 pages,7 figures
