发表机构
College of William and Mary; Tribhuvan University（威廉玛丽学院; 特里布文大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对现有上下文内知识编辑方法难以平衡可靠、泛化、特定目标的问题，提出多目标强化学习算法MO-IKE，在Llama-3.2上多项指标较以往方法显著提升。
AI 中文摘要
大型语言模型（LLMs）功能强大，但受限于静态参数知识，一旦预训练结束便会过时。知识编辑旨在解决这一问题，无需完全重新训练即可更新模型在目标事实上的行为。其中，上下文内知识编辑因无需训练且可直接应用于黑盒LLMs而受到关注。近期基于强化学习（RL）的方法通过调整提示构造以应对数量与质量的权衡，优于固定检索策略。尽管取得了初步成功，但这些方法未能将提示建模为结构化实体，以应对可靠、泛化和特定这三个截然不同且常相互竞争的目标。以往方法大多仅优化单一目标，且仅在提示构造过程的部分环节做出决策，从而忽略了不同目标间的平衡以及演示内容的全局组织。我们提出多目标上下文内知识编辑（MO-IKE），这是一种多目标强化学习算法，将上下文内知识编辑的提示构造形式化为约束马尔可夫决策过程。MO-IKE训练了一个动态检索器，以优化知识编辑中相互竞争的目标，实现更平衡且全局连贯的提示构造。在Llama-3.2上，与以往基于RL的方法相比，MO-IKE将编辑成功率（可靠性）从85.0%提升至92.0%，释义一致性（泛化性）从77%提升至79%，同时保留率（特定性）提高了23.0%。
英文摘要
Large Language Models (LLMs) are powerful but limited by static parametric knowledge that becomes outdated once pretraining ends. Knowledge editing addresses this problem by updating model behavior on target facts without full retraining. In particular, in-context knowledge editing has gained attention because it is training-free and readily applicable to black-box LLMs. Recent reinforcement learning (RL)-based approaches improve over fixed retrieval strategies by adapting prompt construction to the quantity-quality trade-off. Despite initial success, they fail to model the prompt as a structured entity under the distinct and often competing objectives of reliability, generality, and specificity. Previous methods largely optimize a single objective and make decisions over only part of the prompt construction process, thereby overlooking both the balance of different objectives and the global organization of demonstrations. We propose Multi-Objective In-context Knowledge Editing (MO-IKE), a multi-objective RL algorithm that formulates prompt construction for in-context knowledge editing as a Constrained Markov Decision Process. MO-IKE trains a dynamic retriever to optimize competing objectives in knowledge editing, enabling more balanced and globally coherent prompt construction. On Llama-3.2, MO-IKE improves edit success (reliability) from 85.0% to 92.0%, paraphrase consistency (generality) from 77% to 79%, while increasing retention rate (specificity) by 23.0% compared to prior RL-based methods.
CommentsOur work proposes a multi-objective reinforcement learning algorithm that optimizes prompt construction for reliable, generalizable, and specific in-context knowledge-editing
