arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.26663cs.CL
收藏

参数共享使词嵌入偏向输出空间

Weight Tying Biases Token Embeddings Towards the Output Space

EleutherAI;University of California Berkeley(加州大学伯克利分校)

Antonio Lopardo, Avyukth Harish, Catherine Arnett, Akshat Gupta

AI总结:

本文研究了参数共享对词嵌入空间的影响,发现共享矩阵更偏向输出预测而非输入表示,且早期训练中输出梯度主导导致偏差,通过调整梯度可缓解此问题。

摘要:

参数共享,即在输入和输出嵌入矩阵间共享参数,是语言模型设计中的常见做法,但其对学习嵌入空间的影响仍不明确。本文表明,共享的嵌入矩阵更接近输出(解嵌)矩阵而非输入嵌入,表明共享矩阵主要为输出预测而非输入表示所塑造。这种解嵌偏差源于早期训练中输出梯度占主导。通过调优的透镜分析,我们发现这会负面影响早期层的计算,这些计算对残差流的贡献较少。在训练中缩放输入梯度可减少此偏差,提供了梯度不平衡作用的因果证据。这证明了参数共享优化嵌入矩阵以适应输出预测,从而削弱了其在输入表示中的作用。这些结果有助于解释为何参数共享在大规模情况下可能损害性能,并对训练较小的LLMs有影响,因为在这些模型中,嵌入矩阵对总参数数量有显著贡献。

英文摘要:

Weight tying, i.e. sharing parameters between input and output embedding matrices, is common practice in language model design, yet its impact on the learned embedding space remains poorly understood. In this paper, we show that tied embedding matrices align more closely with output (unembedding) matrices than with input embeddings of comparable untied models, indicating that the shared matrix is shaped primarily for output prediction rather than input representation. This unembedding bias arises because output gradients dominate early in training. Using tuned lens analysis, we show this negatively affects early-layer computations, which contribute less effectively to the residual stream. Scaling input gradients during training reduces this bias, providing causal evidence for the role of gradient imbalance. This is mechanistic evidence that weight tying optimizes the embedding matrix for output prediction, compromising its role in input representation. These results help explain why weight tying can harm performance at scale and have implications for training smaller LLMs, where the embedding matrix contributes substantially to total parameter count.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑