arXiv 2504.029042025-11-08cs.CLcs.AIcs.LG

后训练如何重塑大语言模型：关于知识、真实性、拒绝与置信度的机制视角

How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence

University of California, Los Angeles（加州大学洛杉矶分校）；University of Alberta（阿尔伯塔大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Harvard University（哈佛大学）

Hongzhe Du, Weikai Li, Min Cai, Karim Saraipour, Zimin Zhang, Himabindu Lakkaraju, Yizhou Sun, Shichang Zhang

备注：: COLM 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文从知识存储、真实性方向、拒绝方向和置信度四个机制视角对比基础与后训练大语言模型，揭示后训练保留和改变了哪些内部机制，为模型引导和可解释性研究提供依据。

摘要：

后训练对于大语言模型（LLMs）的成功至关重要，它将预训练基础模型转化为更有用且更对齐的后训练模型。尽管已有大量工作研究了后训练算法，并通过模型输出评估后训练模型，但后训练如何在内部重塑大语言模型仍研究不足。在本文中，我们从四个视角对基础模型和后训练大语言模型进行机制层面的比较，以更好地理解后训练效应。我们跨模型家族和数据集的研究发现表明：（1）后训练不会改变事实性知识的存储位置，它在发展新知识表示的同时，对基础模型中的知识表示进行适配；（2）真实性和拒绝都可以在隐藏表示空间中用向量表示。真实性方向在基础模型和后训练模型之间高度相似，并且能够有效地迁移用于干预；（3）拒绝方向在基础模型和后训练模型之间不同，且表现出有限的前向可迁移性；（4）基础模型和后训练模型之间的置信度差异不能归因于熵神经元。我们的研究为后训练过程中被保留和改变的基本机制提供了见解，有助于模型引导等下游任务，并可能有益于未来在可解释性和大语言模型后训练方面的研究。我们的代码已在 https://github.com/HZD01/post-training-mechanistic-analysis 公开。

英文摘要：

Post-training is essential for the success of large language models (LLMs), transforming pre-trained base models into more useful and aligned post-trained models. While plenty of works have studied post-training algorithms and evaluated post-training models by their outputs, it remains understudied how post-training reshapes LLMs internally. In this paper, we compare base and post-trained LLMs mechanistically from four perspectives to better understand post-training effects. Our findings across model families and datasets reveal that: (1) Post-training does not change the factual knowledge storage locations, and it adapts knowledge representations from the base model while developing new knowledge representations; (2) Both truthfulness and refusal can be represented by vectors in the hidden representation space. The truthfulness direction is highly similar between the base and post-trained model, and it is effectively transferable for interventions; (3) The refusal direction is different between the base and post-trained models, and it shows limited forward transferability; (4) Differences in confidence between the base and post-trained models cannot be attributed to entropy neurons. Our study provides insights into the fundamental mechanisms preserved and altered during post-training, facilitates downstream tasks like model steering, and could potentially benefit future research in interpretability and LLM post-training. Our code is publicly available at https://github.com/HZD01/post-training-mechanistic-analysis.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。