揭示无结构知识编辑中的上下文依赖性
Uncovering Context Reliance in Unstructured Knowledge Editing
AI总结：
本文提出COIN框架，通过减少上下文依赖性提升大型语言模型的无结构知识编辑效果。
摘要：
对大型语言模型（LLMs）进行现实世界无结构知识的编辑对于纠正和更新其内部参数化知识至关重要。在本工作中，我们重新审视了下一步标记预测（NTP）作为无结构编辑的候选范式。我们发现上下文依赖性是基于NTP方法的关键失败模式，其中从编辑文本中获得的知识高度依赖于其前文上下文，导致在推理过程中缺少该上下文时出现回忆失败。我们的实证验证表明，在推理过程中前置上下文可以恢复知识回忆。我们进一步理论证明，上下文依赖性是基于梯度的优化的固有结果，这种优化倾向于将获得的知识绑定到特定的聚合上下文表示中。为了解决这一问题，我们提出了一个简单而有效的上下文无关编辑框架（COIN），鼓励模型专注于局部范围内的知识，而不是记忆上下文模式。评估显示，COIN将上下文依赖性降低了45.2%，并在编辑成功率方面超越了强大的基线方法23.6%，突显了缓解上下文依赖性对于稳健编辑的重要性。
英文摘要：
Editing Large language models (LLMs) with real-world, unstructured knowledge is essential for correcting and updating their internal parametric knowledge. In this work, we revisit the fundamental next-token prediction (NTP) as a candidate paradigm for unstructured editing. We identify Context Reliance as a critical failure mode of NTP-based approaches, where knowledge acquired from edited text becomes highly dependent on its preceding context, leading to recall failures when that context is absent during inference. This hypothesis is supported by our empirical validation that prepending context during inference recovers knowledge recall. We further theoretically demonstrate that Context Reliance is an inherent consequence of gradient-based optimization, which tends to bind acquired knowledge to a specific aggregated contextual representation. To address this, we propose a simple yet effective COntext-INdependent editing framework (COIN), encouraging model to focus on knowledge within local scope rather than memorizing contextual patterns. Evaluations show that COIN reduces Context Reliance by 45.2% and outperforms strong baselines by 23.6% in editing success rate, highlighting the vital role of mitigating Context Reliance for robust editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
