GeoEdit:面向大语言模型的几何知识编辑
GeoEdit: Geometric Knowledge Editing for Large Language Models
AI总结:
针对大语言模型知识更新中难以兼顾新知识融入与通用知识保持的问题,提出GeoEdit框架,利用参数更新的几何关系区分神经元,结合方向感知识别、遗忘-学习策略及重要性引导融合,在公开数据集上超越现有方法。
摘要:
定期更新对于保持大语言模型(LLMs)中的知识时效性至关重要。因此,研究者开发了多种模型编辑方法来更新LLM中的特定知识。然而,基于训练的方法往往难以在有效融入新知识的同时保留无关的通用知识。为应对这一挑战,我们提出了一种名为几何知识编辑(GeoEdit)的新框架。GeoEdit利用微调过程中参数更新的几何关系,来区分与新知识更新相关的神经元和与通用知识扰动相关的神经元。通过采用一种方向感知的知识识别方法,我们避免更新与现有知识方向近似正交的神经元,从而保持模型的泛化能力。对于其余神经元,我们对对齐方向整合新旧知识,并对相反方向应用“先遗忘后学习”的编辑策略。此外,我们引入了一种重要性引导的任务向量融合技术,该技术过滤冗余信息并提供自适应的神经元级加权,进一步提升了模型编辑性能。在两个公开数据集上进行的大量实验表明,GeoEdit优于现有的最先进方法。
英文摘要:
Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs). Consequently, various model editing methods have been developed to update specific knowledge within LLMs. However, training-based approaches often struggle to effectively incorporate new knowledge while preserving unrelated general knowledge. To address this challenge, we propose a novel framework called Geometric Knowledge Editing (GeoEdit). GeoEdit utilizes the geometric relationships of parameter updates from fine-tuning to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. By employing a direction-aware knowledge identification method, we avoid updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model's generalization ability. For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a "forget-then-learn" editing strategy for opposite directions. Additionally, we introduce an importance-guided task vector fusion technique that filters out redundant information and provides adaptive neuron-level weighting, further enhancing model editing performance. Extensive experiments on two publicly available datasets demonstrate the superiority of GeoEdit over existing state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。