arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.03434cs.CL
收藏

时间进程机制可解释性:分析大语言模型中组件与知识的演化

Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models

Center for Information and Language Processing, LMU Munich(信息与语言处理中心,慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心)

Ahmad Dawar Hakimi, Ali Modarressi, Philipp Wicke, Hinrich Schütze

AI总结:

本文通过追踪OLMo-7B预训练过程中注意力头和FFNs的角色演化,发现组件从通用走向专业化并存在自适应重利用,且FFNs比注意力头更稳定,揭示了LLMs事实知识形成的机制。

摘要:

理解大语言模型(LLMs)如何获取和存储事实知识,对于提升其可解释性和可靠性至关重要。在本工作中,我们通过跟踪OLMo-7B模型在预训练过程中注意力头和前馈网络(FFNs)的角色变化,分析事实知识表示的演化。我们将这些组件划分为四类角色:通用型、实体型、关系-答案型和事实-答案型,并考察其稳定性和转变。结果表明,LLMs最初依赖广泛的通用组件,随后随着训练推进逐渐专业化。一旦模型能够可靠预测答案,部分组件会被重新调整用途,表明存在一种自适应学习过程。值得注意的是,注意力头表现出最高的更替率。我们还提供证据表明,FFNs在整个训练过程中保持更稳定。此外,我们的探测实验揭示,基于位置的关系比基于名称的关系在训练中更早收敛到高准确率,凸显了任务复杂度如何塑造知识获取动态。这些见解为LLMs中的知识形成提供了机制性视角。

英文摘要:

Understanding how large language models (LLMs) acquire and store factual knowledge is crucial for enhancing their interpretability and reliability. In this work, we analyze the evolution of factual knowledge representation in the OLMo-7B model by tracking the roles of its attention heads and feed forward networks (FFNs) over the course of pre-training. We classify these components into four roles: general, entity, relation-answer, and fact-answer specific, and examine their stability and transitions. Our results show that LLMs initially depend on broad, general-purpose components, which later specialize as training progresses. Once the model reliably predicts answers, some components are repurposed, suggesting an adaptive learning process. Notably, attention heads display the highest turnover. We also present evidence that FFNs remain more stable throughout training. Furthermore, our probing experiments reveal that location-based relations converge to high accuracy earlier in training than name-based relations, highlighting how task complexity shapes acquisition dynamics. These insights offer a mechanistic view of knowledge formation in LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑