arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09338cs.CL
收藏

稳健编辑的关键:从理论洞见到实践进展

Keys to Robust Edits: from Theoretical Insights to Practical Advances

Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖先进研究院技术研究所);Shanghai AI Lab(上海人工智能实验室)

Jianhao Yan, Futing Wang, Yun Luo, Yafu Li, Yue Zhang

AI总结:

针对定位-编辑方法在长上下文和改写查询中稳健性不足的问题,提出稳健编辑路径(REP),通过解耦编辑键和对比学习平衡稳健性与特异性,在多种模型和数据集上显著提升稳健性成功率。

摘要:

大型语言模型(LLMs)由于参数记忆中存在冲突或过时信息,难以维持准确的知识。虽然定位-编辑方法解决了这一问题,但它们对模型内部表示的依赖导致在长上下文推理和改写查询中稳健性失败。我们识别出定位-编辑方法的一个根本局限:现有的语义键(用于记忆定位)无法同时满足稳健性(上下文不变的激活)和特异性(精确的知识区分)。通过理论误差界分析,我们建立了有效编辑的正式标准。我们的解决方案引入了稳健编辑路径(REP),一个即插即用模块,它:(1)将编辑键从模型原生表示中解耦;(2)通过对比学习动态调整键以实现稳健性-特异性平衡。在多种编辑方法(ROME/MEMIT/R-ROME/EMMET)、现有LLMs(LLaMA2、QWen、Mistral)和数据集(CounterFact、ZsRE)上的广泛实验表明,REP在稳健性测试上的成功率提高了最多66.4%,同时保持成功率不受影响。我们的代码可在https://github.com/ElliottYan/RobustKeyEdit获取。

英文摘要:

Large language models (LLMs) struggle with maintaining accurate knowledge due to conflicting/outdated parametric memories. While locate-and-edit methods address this, their reliance on models' internal representations leads to robustness failures in long-context reasoning and paraphrased queries. We identify a fundamental limitation of locate-and-edit methods: existing semantic keys (for memory localization) cannot simultaneously satisfy robustness (context-invariant activation) and specificity (precise knowledge discrimination). Through theoretical error-bound analysis, we establish formal criteria for effective editing. Our solution introduces \textit{Robust Edit Pathway (REP)}, a plug-and-play module that: (1) disentangles editing keys from native model representations; (2) dynamically adjusts keys via contrastive learning to achieve robustness-specificity balance. Extensive experiments across various editing methods (ROME/MEMIT/R-ROME/EMMET), existing LLMs (LLaMA2, QWen, Mistral), and datasets (CounterFact, ZsRE) show that REP improves success rate over robustness tests by up-to 66.4\% while maintaining the success rate unaffected. Our code can be found at https://github.com/ElliottYan/RobustKeyEdit .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑