arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.20207cs.LGcs.AI
收藏

黄金层与寻找它们的位置:通过层梯度分析改进大型语言模型的知识编辑

Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysis

University of South Florida(佛罗里达州立大学);Brandeis University(布兰迪大学)

Shrestha Datta, Hongfu Liu, Anshuman Chhabra

AI总结:

本文提出通过层梯度分析(LGA)识别高效知识编辑的黄金层,验证其在不同数据集和模型类型中的有效性。

摘要:

在大型语言模型(LLMs)中,知识编辑旨在更新特定查询的预测以达到目标,同时保持其他输入的行为。此过程通常分为两个阶段:确定要编辑的层和执行参数更新。本文假设存在固定的黄金层,可实现接近最优的编辑性能。通过比较黄金层与地面真实样本最优层,我们提供了实证证据。此外,我们展示黄金层可通过代理数据集可靠识别,并在未见过的测试集查询中泛化有效。最后,我们提出了一种新的方法,即层梯度分析(LGA),通过梯度归因高效估计黄金层,避免了多次编辑运行中的大量试错。在多个基准数据集上的广泛实验证明了LGA方法在不同LLM类型和各种知识编辑方法中的有效性和鲁棒性。

英文摘要:

Knowledge editing in Large Language Models (LLMs) aims to update the model's prediction for a specific query to a desired target while preserving its behavior on all other inputs. This process typically involves two stages: identifying the layer to edit and performing the parameter update. Intuitively, different queries may localize knowledge at different depths of the model, resulting in different sample-wise editing performance for a fixed editing layer. In this work, we hypothesize the existence of fixed golden layers that can achieve near-optimal editing performance similar to sample-wise optimal layers. To validate this hypothesis, we provide empirical evidence by comparing golden layers against ground-truth sample-wise optimal layers. Furthermore, we show that golden layers can be reliably identified using a proxy dataset and generalize effectively to unseen test set queries across datasets. Finally, we propose a novel method, namely Layer Gradient Analysis (LGA) that estimates golden layers efficiently via gradient-attribution, avoiding extensive trial-and-error across multiple editing runs. Extensive experiments on several benchmark datasets demonstrate the effectiveness and robustness of our LGA approach across different LLM types and various knowledge editing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑