大型Transformer模型在许多自然语言任务中取得了令人印象深刻的性能。特别地，基于Transformer的语言模型已经被证明在以其大量参数对事实知识进行编码方面具有很强的能力。虽然提高Transformers的记忆和概括能力的任务已经得到了广泛的研究，但如何使transformers忘记特定的旧事实并记住新事实还不太清楚。在本文中，我们提出了一个新的任务\emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}。此任务在许多情况下都很有用，例如更新陈旧的知识，保护隐私，以及消除存储在模型中的意外偏见。我们对几种方法进行了基准测试，这些方法在这项任务上提供了自然的基线性能。这导致了Transformer模型的关键组件的发现，这些组件对于知识修改特别有效。这项工作还提供了不同的训练阶段（如预训练和微调）对记忆和知识修改的作用的见解。
英文摘要：
Large Transformer models have achieved impressive performance in many natural language tasks. In particular, Transformer based language models have been shown to have great capabilities in encoding factual knowledge in their vast amount of parameters. While the tasks of improving the memorization and generalization of Transformers have been widely studied, it is not well known how to make transformers forget specific old facts and memorize new ones. In this paper, we propose a new task of \emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}. This task is useful in many scenarios, such as updating stale knowledge, protecting privacy, and eliminating unintended biases stored in the models. We benchmarked several approaches that provide natural baseline performances on this task. This leads to the discovery of key components of a Transformer model that are especially effective for knowledge modifications. The work also provides insights into the role that different training phases (such as pretraining and fine-tuning) play towards memorization and knowledge modification.
Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers
CSE, HKUST（香港科技大学计算机科学与工程系）；IIIS, Tsinghua University（清华大学交叉信息研究院）；ILCC, University of Edinburgh（爱丁堡大学语言、语言学与文化学院）；College of Computer Science, Sichuan University（四川大学计算机学院）
MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training
Faculty of Computer Science and Mathematics, University of Passau, Germany（计算机科学与数学学院，帕绍大学）；Taub Faculty for Computer Science, Technion - Israel Institute of Technology, Israel（计算机科学系，技术ion-以色列理工学院）
Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；University of Electronic Science and Technology of China（电子科技大学）
Causal Language Control in Multilingual Transformers via Sparse Feature Steering
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Maryland, College Park（马里兰大学学院公园分校）；Barnard College（巴纳德学院）；Algoverse AI Research（Algoverse人工智能研究）；Meta FAIR Lab（Meta FAIR实验室）
