MEMIT-Merge:解决MEMIT在同主题批量编辑中的键值冲突问题
MEMIT-Merge: Addressing MEMIT's Key-Value Conflicts in Same-Subject Batch Editing for LLMs
AI总结:
针对MEMIT批量编辑中同主题键值冲突导致性能下降的问题,提出MEMIT-Merge方法,通过合并共享主题的值计算过程,将编辑成功率从约50%提升至90%以上。
摘要:
随着大型语言模型的规模不断扩大,无需完全重新训练即可修改模型内部知识的知识编辑技术受到了广泛关注。MEMIT作为一种突出的批量编辑算法,因其能够进行大规模知识修改而脱颖而出。然而,我们发现当处理包含多个共享同一主题的编辑的批次时,MEMIT的编辑效果显著下降。我们的分析表明,这源于MEMIT的键值建模框架:相同的键(源自共享主题)被迫表示不同的值(对应不同的知识),导致编辑过程中出现更新冲突。针对这一问题,我们提出了MEMIT-Merge,一种增强方法,它合并了共享同一主题的事实的值计算过程,有效解决了同主题批量编辑场景中的性能下降问题。实验结果表明,当MEMIT在较大批量大小下的编辑成功率降至约50%时,MEMIT-Merge保持了超过90%的成功率,展现出对主题实体冲突的显著鲁棒性。代码可在https://github.com/NUSTM/MEMIT-Merge获取。
英文摘要:
As large language models continue to scale up, knowledge editing techniques that modify models' internal knowledge without full retraining have gained significant attention. MEMIT, a prominent batch editing algorithm, stands out for its capability to perform mass knowledge modifications. However, we uncover that MEMIT's editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject. Our analysis reveals this stems from MEMIT's key value modeling framework: identical keys (derived from the shared subject) are forced to represent different values (corresponding to different knowledge), resulting in update conflicts during editing. Addressing this issue, we propose MEMIT-Merge, an enhanced approach that merges value computation processes for facts sharing the same subject, effectively resolving the performance degradation in samesubject batch editing scenarios. Experimental results demonstrate that when MEMIT's edit success rate drops to around 50% at larger batch sizes, MEMIT-Merge maintains a success rate exceeding 90%, showcasing remarkable robustness to subject entity collisions. The code is available at https://github.com/NUSTM/ MEMIT-Merge.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。