通过训练动力学理论理解“逆转诅咒”
Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics
AI总结：
本文利用梯度下降训练动力学分析双线性模型与单层transformer，证明“逆转诅咒”源于权重更新的不对称，并通过多层transformer实验验证理论。
摘要：
自回归大语言模型（LLM）在解决许多复杂推理任务方面表现出令人瞩目的能力，却在一些简单逻辑推理任务上遇到困难，例如逆向搜索：当模型在“$A \to B$”（例如“Tom是John的家长”）上接受训练后，LLM在推理时无法直接得出“$B \gets A$”（例如“John是Tom的孩子”），即使这两个句子在语义上完全相同，这一现象被称为“逆转诅咒”。本文通过（随机）梯度下降的训练动力学，对两类自回归模型中的逆转诅咒进行理论分析：（1）可视为单层transformer简化形式的双线性模型；（2）在特定假设下的单层transformer。我们的分析表明，对于这两种模型，逆转诅咒都是（有效）模型权重“不对称”的结果，即训练期间从词元$A$到词元$B$的权重增加并不必然导致从$B$到$A$的权重增加；这是由特定损失函数选择下的训练动力学以及模型参数的优化空间造成的。此外，我们的分析可以自然应用于思维链（COT）等其他逻辑推理任务，提供了不同于以往关注表达能力的研究的新视角。最后，我们在不同设置下对多层transformer进行实验，以验证我们的理论。代码可在https://github.com/marlo-z/reversal_curse_analysis/获取。
英文摘要：
Auto-regressive large language models (LLMs) show impressive capacities to solve many complex reasoning tasks while struggling with some simple logical reasoning tasks such as inverse search: when trained on '$A \to B$' (e.g., 'Tom is the parent of John'), LLM fails to directly conclude '$B \gets A$' (e.g., 'John is the child of Tom') during inference even if the two sentences are semantically identical, which is known as the 'reversal curse'. In this paper, we theoretically analyze the reversal curse via the training dynamics of (stochastic) gradient descent for two auto-regressive models: (1) a bilinear model that can be viewed as a simplification of a one-layer transformer; (2) one-layer transformers under certain assumptions. Our analysis reveals that for both models, the reversal curse is a consequence of the (effective) model weights 'asymmetry', i.e., the increase of weights from a token $A$ to token $B$ during training does not necessarily cause the increase of the weights from $B$ to $A$, which is caused by the training dynamics under certain choice of loss function and the optimization space of model parameters. Moreover, our analysis can be naturally applied to other logical reasoning tasks such as chain-of-thought (COT), which provides a new perspective different from previous work that focuses on expressivity. Finally, we conduct experiments to validate our theory on multi-layer transformers under different settings. Our code is available at https://github.com/marlo-z/reversal_curse_analysis/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
