语言模型的物理：第3.1部分，知识存储与提取
Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
AI总结：
本文通过受控传记数据集研究LLM知识提取能力，发现知识需在预训练中充分增强（如释义、打乱、翻译）才能可靠提取，否则即使指令微调也无法提取，并提出重写预训练数据及增加指令微调数据等工业建议。
摘要：
大型语言模型（LLMs）能够存储大量的世界知识，通常可以通过问答（例如，“亚伯拉罕·林肯的生日是什么？”）来提取。然而，它们是基于训练期间接触到类似问题（即作弊）来回答这些问题，还是通过真正学习从维基百科等来源提取知识？在本文中，我们使用一个受控的传记数据集来研究这个问题。我们发现模型提取知识的能力与训练数据的各种多样性度量之间存在很强的相关性。本质上，为了可靠地提取知识，必须在预训练期间对其进行充分的增强（例如，通过释义、句子打乱、翻译）。如果没有这种增强，知识可能被记忆但无法提取，导致0%的准确率，无论后续进行何种指令微调。为了理解为什么会发生这种情况，我们采用（近乎）线性探针来证明观察到的相关性与模型内部编码知识的方式之间存在紧密联系——即知识是线性编码在实体名称的隐藏嵌入中，还是分布在训练文本中的其他标记嵌入中。本文为工业界的LLM预训练提供了几个关键建议：（1）使用小型辅助模型重写预训练数据，以提供知识增强，以及（2）在预训练阶段纳入更多的指令微调数据，以免为时过晚。
英文摘要：
Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering (e.g., "What is Abraham Lincoln's birthday?"). However, do they answer such questions based on exposure to similar questions during training (i.e., cheating), or by genuinely learning to extract knowledge from sources like Wikipedia? In this paper, we investigate this issue using a controlled biography dataset. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data. $\textbf{Essentially}$, for knowledge to be reliably extracted, it must be sufficiently augmented (e.g., through paraphrasing, sentence shuffling, translations) $\textit{during pretraining}$. Without such augmentation, knowledge may be memorized but not extractable, leading to 0% accuracy, regardless of subsequent instruction fine-tuning. To understand why this occurs, we employ (nearly) linear probing to demonstrate a strong connection between the observed correlation and how the model internally encodes knowledge -- whether it is linearly encoded in the hidden embeddings of entity names or distributed across other token embeddings in the training text. This paper provides $\textbf{several key recommendations for LLM pretraining in the industry}$: (1) rewrite the pretraining data -- using small, auxiliary models -- to provide knowledge augmentation, and (2) incorporate more instruction-finetuning data into the pretraining stage before it becomes too late.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
