智慧在于知道什么不说：通过注意力转移实现无幻觉的LLM去学习
Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting
AI总结：
本文提出注意力转移框架，通过保留上下文抑制和抗幻觉响应塑造，提升LLM去学习效果，在保持性能的同时减少幻觉响应。
摘要：
计算能力的提升和AI辅助决策的必要性推动了大型语言模型（LLMs）的广泛应用。与此同时，LLMs潜在保留敏感数据引发了对机器去学习的研究。然而，现有去学习方法面临关键难题：激进去学习损害模型效用，而保守策略保持效用但有幻觉风险。为解决此问题，我们引入了注意力转移（AS）框架进行选择性去学习。AS由两个设计目标驱动：（1）上下文保留抑制，通过减弱对事实性token的注意力而不破坏LLM的语言结构；（2）抗幻觉响应塑造，通过避免在查询去学习内容时生成伪造完成。AS通过两个注意力层面干预实现这些目标：重要性感知抑制应用于去学习集以减少对记忆知识的依赖，以及注意力引导保留增强以加强保留数据集中语义关键token的注意力以减轻意外退化。这两个组件通过双损失目标联合优化，形成一个软边界，使去学习局部化的同时在表示叠加下保留无关知识。实验结果表明，AS在状态-of-the-art去学习方法上提升了性能保留，达到ToFU基准的15%和TDEC基准的10%的准确率，同时保持竞争性的无幻觉去学习效果。与现有方法相比，AS在去学习效果、泛化能力和响应可靠性之间表现出更优的平衡。
英文摘要：
The increase in computing power and the necessity of AI-assisted decision-making boost the growing application of large language models (LLMs). Along with this, the potential retention of sensitive data of LLMs has spurred increasing research into machine unlearning. However, existing unlearning approaches face a critical dilemma: Aggressive unlearning compromises model utility, while conservative strategies preserve utility but risk hallucinated responses. This significantly limits LLMs' reliability in knowledge-intensive applications. To address this, we introduce a novel Attention-Shifting (AS) framework for selective unlearning. AS is driven by two design objectives: (1) context-preserving suppression that attenuates attention to fact-bearing tokens without disrupting LLMs' linguistic structure; and (2) hallucination-resistant response shaping that discourages fabricated completions when queried about unlearning content. AS realizes these objectives through two attention-level interventions, which are importance-aware suppression applied to the unlearning set to reduce reliance on memorized knowledge and attention-guided retention enhancement that reinforces attention toward semantically essential tokens in the retained dataset to mitigate unintended degradation. These two components are jointly optimized via a dual-loss objective, which forms a soft boundary that localizes unlearning while preserving unrelated knowledge under representation superposition. Experimental results show that AS improves performance preservation over the state-of-the-art unlearning methods, achieving up to 15% higher accuracy on the ToFU benchmark and 10% on the TDEC benchmark, while maintaining competitive hallucination-free unlearning effectiveness. Compared to existing methods, AS demonstrates a superior balance between unlearning effectiveness, generalization, and response reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
