发表机构
Georgia Institute of Technology; University of Pennsylvania; University at Albany; University of Tennessee（佐治亚理工学院; 宾夕法尼亚大学; 奥尔巴尼大学; 田纳西大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对非结构化知识编辑中上下文依赖导致原子事实回忆失败的问题，提出FOVEATED框架，通过随机扰动RoPE位置构建聚焦视图，在编辑时应用、推理时移除，显著提升五个编辑器在两个骨干和三个基准上的事实回忆性能。
AI 中文摘要
大型语言模型（LLMs）日益充当事实知识的通用接口，但其参数不会自动反映预训练后发生变化的信息。知识编辑（KE）通过修改选定的知识并保留无关知识和通用能力，提供了一种针对昂贵重训的替代方案。传统KE使用结构化事实三元组，而非结构化KE（UKE）使用包含多个事实的自由文本段落。然而，现有的UKE编辑器表现出一种称为上下文依赖的失败模式：编辑后的LLM通常能复现编辑段落，但在没有原始段落上下文的情况下，无法可靠地回忆其个别事实。我们识别出在标准段落级编辑目标下，上下文引起的难度低估问题：后续事实获得越来越丰富的真实上下文，因此初始损失较低，使它们看起来更容易学习。为此，我们提出了FOVEATED，一个即插即用框架，通过随机移动其前文上下文的键所分配的旋转位置编码（RoPE）位置，为每个句子构建聚焦视图。该扰动在编辑期间应用，之后移除，在推理时保持模型的原生位置编码不变。我们为直接优化和定位-然后-编辑两种编辑器实例化了FOVEATED。我们从理论上分析了FOVEATED如何抵消上下文引起的难度低估，并在五个KE编辑器、两个LLM骨干网络和三个基准上实证证明了一致的改进。
英文摘要
Large language models (LLMs) increasingly serve as general-purpose interfaces to factual knowledge, but their parameters do not automatically reflect information that changes after pretraining. Knowledge editing (KE) provides a targeted alternative to costly retraining by modifying selected knowledge and preserving unrelated knowledge and general capabilities. Conventional KE uses structured factual triples, whereas unstructured KE (UKE) uses free-form passages containing multiple facts. Nonetheless, existing UKE editors exhibit a failure mode known as context reliance: edited LLMs can often reproduce the editing passage but fail to reliably recall its individual facts without the original passage context. We identify context-induced difficulty underestimation under the standard passage-level editing objective: later facts receive increasingly rich ground-truth context and consequently incur lower initial losses, making them appear easier to learn. In response, we propose FOVEATED, a plug-and-play framework that constructs focused views of each sentence by randomly shifting the Rotary Position Embedding (RoPE) positions assigned to the keys of its preceding context. The perturbation is applied during editing and removed afterward, leaving the model's native positional encoding unchanged at inference time. We instantiate FOVEATED for both direct-optimization and locate-then-edit editors. We theoretically analyze how FOVEATED counteracts context-induced difficulty underestimation and empirically demonstrate consistent improvements across five KE editors, two LLM backbones, and three benchmarks.
CommentsThe first two authors contributed equally
