神经数学问题求解器中的中间值跟踪和操作
Tracing and Manipulating Intermediate Values in Neural Math Problem Solvers
摘要:
语言模型如何处理需要多个推理步骤的复杂输入还没有得到很好的理解。先前的研究表明,关于这些输入的中间值的信息可以从模型的激活中提取出来,但目前还不清楚这些信息是在哪里编码的,以及这些信息是否确实在推理过程中使用。我们介绍了一种方法来分析如何一个Transformer模型处理这些输入,专注于简单的算术问题和它们的中间值。为了跟踪中间值的信息被编码的位置,我们使用主成分分析(PCA)来测量中间值和模型激活之间的相关性。然后,我们通过操纵模型权重进行因果干预。这种干预表明,通过跟踪确定的权重不仅与中间值相关,而且与模型预测有因果关系。我们的研究结果表明,该模型具有局部性的某些中间值,这是有用的,以提高模型的可解释性。
英文摘要:
How language models process complex input that requires multiple steps of inference is not well understood. Previous research has shown that information about intermediate values of these inputs can be extracted from the activations of the models, but it is unclear where that information is encoded and whether that information is indeed used during inference. We introduce a method for analyzing how a Transformer model processes these inputs by focusing on simple arithmetic problems and their intermediate values. To trace where information about intermediate values is encoded, we measure the correlation between intermediate values and the activations of the model using principal component analysis (PCA). Then, we perform a causal intervention by manipulating model weights. This intervention shows that the weights identified via tracing are not merely correlated with intermediate values, but causally related to model predictions. Our findings show that the model has a locality to certain intermediate values, and this is useful for enhancing the interpretability of the models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。