arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20748cs.LGcs.AI
收藏

解释语言模型的参数

Interpreting Language Model Parameters

Goodfire(古德法尔);MATS

Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, Lee Sharkey

AI总结:

本研究针对语言模型参数的机制可解释性问题,提出对抗参数分解(VPD)方法,将参数分解为简单子组件,实现新型电路分析,为后续机制分析提供基础。

摘要:

机制可解释性旨在揭示神经网络如何利用其参数实现令人印象深刻的神经算法。尽管已有研究发现了网络中间表示中存在大量结构,但对于网络的参数和非线性如何对这些表示执行计算,相关理解进展甚微。在本研究中,我们提出了一种方法,通过将语言模型的参数分解为仅实现模型所学算法一小部分的子组件,同时要求仅需这些子组件的一小部分即可解释网络在任意输入上的行为,从而推动了这一理解。该方法名为对抗参数分解(adVersarial Parameter Decomposition,VPD),其优化目标是将神经网络参数分解为简单子组件,即使在对大量子组件进行消融(包括为破坏行为而对抗选择的消融)时,仍能保留网络的输入-输出行为。这鼓励学习能为网络行为提供简短、机制上忠实描述的子组件,这些子组件应能适当地聚合为对网络所学算法的更全局描述。我们研究了这些参数子组件之间的交互序列如何在特定输入上产生网络输出,从而实现了一种新型的“电路”分析。尽管要加深对神经网络如何利用其参数计算行为的理解仍需更多工作,但本研究提出了一种方法,可识别出一小组简单、机制上忠实的子组件,为进一步的机制分析奠定基础。

英文摘要:

Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in the intermediate representations that networks use, little progress has been made to understand how the parameters and nonlinearities of networks perform computations on those representations. In this work, we present a method that brings us closer to this understanding by decomposing a language model's parameters into subcomponents that each implement only a small part of the model's learned algorithm, while simultaneously requiring only a small fraction of those subcomponents to account for the network's behavior on any input. The method, adVersarial Parameter Decomposition (VPD), optimizes for decompositions of neural network parameters into simple subcomponents that preserve the network's input-output behavior even when many subcomponents are ablated, including under ablations that are adversarially selected to destroy behavior. This encourages learning subcomponents that provide short, mechanistically faithful descriptions of the network's behavior that should aggregate appropriately into more global descriptions of the network's learned algorithm. We study how sequences of interactions between these parameter subcomponents produce the network's output on particular inputs, enabling a new kind of 'circuit' analysis. While more work remains to be done to deepen our understanding of how neural networks use their parameters to compute their behavior, our work suggests an approach to identify a small set of simple, mechanistically faithful subcomponents on which further mechanistic analysis can be based.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑