PolySAE: 通过多项式解码建模稀疏自编码器中的特征交互
PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding
AI总结：
提出PolySAE，在稀疏自编码器解码器中引入高阶项以建模特征交互，通过低秩张量分解在共享投影子空间上捕获成对和三元特征交互，在保持可解释性的同时提升探测F1约8%，并产生与共现频率无关的组合结构。
摘要：
稀疏自编码器（SAE）通过将激活分解为字典原子的稀疏组合来解释神经网络表示。然而，SAE假设特征通过线性重建相加组合，这种假设无法捕捉组合结构：线性模型无法区分“Starbucks”是由“star”和“coffee”特征的组合还是仅由它们的共现产生。这迫使SAE为复合概念分配整体特征，而不是将其分解为可解释的组成部分。我们引入了PolySAE，它通过高阶项扩展SAE解码器以建模特征交互，同时保留对可解释性至关重要的线性编码器。通过在共享投影子空间上进行低秩张量分解，PolySAE以较小的参数开销（GPT2上为3%）捕获成对和三元特征交互。在四个语言模型和三个SAE变体上，PolySAE在保持可比重建误差的同时，探测F1平均提升约8%，并产生类别条件特征分布之间2-10倍更大的Wasserstein距离。关键的是，学习到的交互权重与共现频率的相关性可忽略不计（r = 0.06，而SAE特征协方差为r = 0.82），表明多项式项捕获了很大程度上独立于表面统计的组合结构。最后，学习到的交互方向因果性地将模型输出引导向相应的组合语义。
英文摘要：
Sparse autoencoders (SAEs) interpret neural network representations by decomposing activations into sparse combinations of dictionary atoms. However, SAEs assume features combine additively through linear reconstruction, an assumption that cannot capture compositional structure: linear models cannot distinguish whether ''Starbucks'' arises from the composition of ''star'' and ''coffee'' features or merely their co-occurrence. This forces SAEs to allocate monolithic features for compound concepts rather than decomposing them into interpretable constituents. We introduce PolySAE, which extends the SAE decoder with higher-order terms to model feature interactions while preserving the linear encoder essential for interpretability. Through low-rank tensor factorization on a shared projection subspace, PolySAE captures pairwise and triple feature interactions with small parameter overhead (3% on GPT2). Across four language models and three SAE variants, PolySAE achieves an average improvement of $\sim$8% in probing F1 while maintaining comparable reconstruction error, and produces 2--10$\times$ larger Wasserstein distances between class-conditional feature distributions. Critically, learned interaction weights exhibit negligible correlation with co-occurrence frequency ($r = 0.06$ vs $r = 0.82$ for SAE feature covariance), suggesting that polynomial terms capture compositional structure largely independent of surface statistics. Finally, the learned interaction directions causally steer model outputs toward the corresponding compositional semantics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
