初始化对Transformer是通过推理还是记忆拟合复合函数至关重要
Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
AI总结：
该研究发现参数初始化尺度决定Transformer学习推理型还是记忆型组合解，揭示了两类解的不同机制与低复杂度偏差的作用，提出初始化率$γ$作为可调超参数，为调节模型推理与记忆能力提供了新视角。
摘要：
Transformer在各类任务中展现出令人瞩目的能力，但它们在组合问题上的表现仍是一个存在争议的话题。本研究探究了Transformer在未见过的组合任务上的行为机制。我们发现，参数初始化尺度在决定模型学习推理型（基于推理的）解还是对称型（基于记忆的）解方面起着关键作用：推理型解能捕捉底层的组合基元，而对称型解只是记忆映射关系，并未理解组合结构。通过分析模型内部的信息流和向量表示，我们揭示了这两类解背后的不同机制。我们进一步发现，推理型（基于推理的）解呈现出低复杂度偏差，我们推测这是使其能够为单个锚点学习独立映射的关键因素。我们在多个真实世界数据集上验证了结论。本研究发现为初始化尺度在调节推理与记忆能力中的作用提供了宝贵见解，我们还提出初始化率$γ$可作为通用深度学习框架中便于调节的超参数，其中$1/d_{\text{in}}^γ$是输入神经元数为$d_{\text{in}}$的层的参数标准差。
英文摘要：
Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the mechanisms of how transformers behave on unseen compositional tasks. We discover that the parameter initialization scale plays a critical role in determining whether the model learns inferential (reasoning-based) solutions, which capture the underlying compositional primitives, or symmetric (memory-based) solutions, which simply memorize mappings without understanding the compositional structure. By analyzing the information flow and vector representations within the model, we reveal the distinct mechanisms underlying these solution types. We further find that inferential (reasoning-based) solutions exhibit low complexity bias, which we hypothesize is a key factor enabling them to learn individual mappings for single anchors. We validate our conclusions on various real-world datasets. Our findings provide valuable insights into the role of initialization scale in tuning the reasoning and memorizing ability and we propose the initialization rate $γ$ to be a convenient tunable hyper-parameter in common deep learning frameworks, where $1/d_{\mathrm{in}}^γ$ is the standard deviation of parameters of the layer with $d_{\mathrm{in}}$ input neurons.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
