SWEA：通过主体词嵌入改变来更新大语言模型中的事实知识
SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering
AI总结：
针对局部模型编辑计算昂贵、匹配不可靠且扰动参数结构的问题，提出可拆卸可扩展的 SWEA⊕OS，通过 token 级主体词嵌入改变与优化-抑制融合实现事实知识更新，并在 CounterFact、zsRE 和 RippleEdits 上达到 SOTA。
摘要：
大语言模型（LLM）的通用能力使其成为各类 AI 应用的基础设施，但更新其内部知识需要大量资源。近年来，模型编辑作为一种高效更新 LLM 少量知识的有前景技术受到广泛关注。尤其是直接更新模型参数的局部编辑方法，已被证明适合更新少量知识。局部编辑方法通过计算最小二乘闭式解来更新权重，并在推理时通过向量级匹配识别被编辑的知识，取得了可观结果。然而，这些方法仍需大量时间和资源才能完成计算；此外，向量级匹配缺乏可靠性，而且此类更新会破坏模型参数的原有组织结构。为解决这些问题，我们提出一种可拆卸、可扩展的主体词嵌入改变（Subject Word Embedding Altering，SWEA）框架，通过 token 级匹配寻找编辑嵌入，并将其加入 Transformer 输入中的主体词嵌入。为获得这些编辑嵌入，我们提出先优化后抑制的融合方法：首先针对编辑目标优化可学习的嵌入向量，随后抑制知识嵌入维度（Knowledge Embedding Dimensions，KEDs），以得到最终编辑嵌入。据此，我们提出用于编辑 LLM 中事实知识的 SWEA⊕OS 方法。我们在 CounterFact 和 zsRE 数据集上证明 SWEA⊕OS 具有整体最先进（SOTA）性能。为进一步验证 SWEA⊕OS 在编辑知识方面的推理能力，我们在更复杂的 RippleEdits 基准上进行评估。结果表明，SWEA⊕OS 具备 SOTA 推理能力。
英文摘要：
The general capabilities of large language models (LLMs) make them the infrastructure for various AI applications, but updating their inner knowledge requires significant resources. Recent model editing is a promising technique for efficiently updating a small amount of knowledge of LLMs and has attracted much attention. In particular, local editing methods, which directly update model parameters, are proven suitable for updating small amounts of knowledge. Local editing methods update weights by computing least squares closed-form solutions and identify edited knowledge by vector-level matching in inference, which achieve promising results. However, these methods still require a lot of time and resources to complete the computation. Moreover, vector-level matching lacks reliability, and such updates disrupt the original organization of the model's parameters. To address these issues, we propose a detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching and adds them to the subject word embeddings in Transformer input. To get these editing embeddings, we propose optimizing then suppressing fusion method, which first optimizes learnable embedding vectors for the editing target and then suppresses the Knowledge Embedding Dimensions (KEDs) to obtain final editing embeddings. We thus propose SWEA$\oplus$OS method for editing factual knowledge in LLMs. We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$\oplus$OS on the CounterFact and zsRE datasets. To further validate the reasoning ability of SWEA$\oplus$OS in editing knowledge, we evaluate it on the more complex RippleEdits benchmark. The results demonstrate that SWEA$\oplus$OS possesses SOTA reasoning ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
