神经网络与Transformer中的傅里叶电路：以多输入模算术为例
Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
AI总结：
本文以多输入模加法为案例，从理论上刻画了单隐藏层神经网络和单层Transformer学习到的傅里叶特征，并证明在足够宽度下网络通过间隔最大化获得最大L2,k+1间隔，从而揭示其采用特定计算策略的内在机制。
摘要：
在不断演进的机器学习领域中，一个关键挑战在于解读神经网络和Transformer所利用的内部表示。基于近期在理解网络如何执行不同目标函数方面取得的进展，本研究致力于探索网络采用特定计算策略的根本原因。我们将关注点集中在涉及$k$个输入的模加法这一复杂代数学习任务上。我们的研究对风格化单隐藏层神经网络和单层Transformer在解决该任务时所学习的特征进行了详尽的分析表征。我们理论框架的一个基石是阐明间隔最大化原则如何塑造单隐藏层神经网络所采用的特征。令$p$表示模数，$D_p$表示具有$k$个输入的模算术数据集，$m$表示网络宽度。我们证明，当神经元数量满足$m \geq 2^{2k-2} \cdot (p-1)$时，这些网络在数据集$D_p$上达到最大$L_{2,k+1}$间隔。此外，我们确定每个隐藏层神经元都与特定的傅里叶频谱对齐，这对解决模加法问题至关重要。通过将我们的发现与类似研究的经验观察相关联，我们有助于更深入地理解神经网络的内在计算机制。此外，我们在单层Transformer的注意力矩阵中观察到类似的计算机制。我们的工作在揭示其运行复杂性方面迈出了重要一步，特别是在复杂代数任务领域。
英文摘要：
In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the internal representations harnessed by neural networks and Transformers. Building on recent progress toward comprehending how networks execute distinct target functions, our study embarks on an exploration of the underlying reasons behind networks adopting specific computational strategies. We direct our focus to the complex algebraic learning task of modular addition involving $k$ inputs. Our research presents a thorough analytical characterization of the features learned by stylized one-hidden layer neural networks and one-layer Transformers in addressing this task. A cornerstone of our theoretical framework is the elucidation of how the principle of margin maximization shapes the features adopted by one-hidden layer neural networks. Let $p$ denote the modulus, $D_p$ denote the dataset of modular arithmetic with $k$ inputs and $m$ denote the network width. We demonstrate that a neuron count of $ m \geq 2^{2k-2} \cdot (p-1) $, these networks attain a maximum $ L_{2,k+1} $-margin on the dataset $ D_p $. Furthermore, we establish that each hidden-layer neuron aligns with a specific Fourier spectrum, integral to solving modular addition problems. By correlating our findings with the empirical observations of similar studies, we contribute to a deeper comprehension of the intrinsic computational mechanisms of neural networks. Furthermore, we observe similar computational mechanisms in attention matrices of one-layer Transformers. Our work stands as a significant stride in unraveling their operation complexities, particularly in the realm of complex algebraic tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
