arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.03571cs.AI
收藏

选择性遗忘用于大推理模型

Selective Forgetting for Large Reasoning Models

Iowa State University(爱荷华州立大学)

Tuan Le, Wei Qian, Mengdi Huai

AI总结:

本文提出一种选择性遗忘框架,通过多语言模型分析推理轨迹,识别并替换敏感内容,以保护一般推理能力,同时解决大推理模型的知识泄露问题。

摘要:

大型推理模型(LRMs)在生成最终答案前会生成结构化的推理轨迹(CoTs),使其特别容易通过中间推理步骤泄露知识。然而,训练数据中敏感信息的存储,如版权和私人内容,已引起伦理和法律问题。为了解决这些问题,选择性遗忘(也称为机器反向学习)已成为LRMs的潜在解决方案。然而,现有的反向学习方法主要针对最终答案,遗忘后可能会降低LRMs的整体推理能力。此外,直接对整个CoTs应用反向学习可能会损害一般推理能力。LRMs反向学习的关键挑战在于在保持一般推理能力的同时精确删除目标知识。为此,本文提出了一种新的LRMs反向学习框架,该框架能够选择性地删除敏感推理组件,同时保持一般推理能力。我们的方法利用多个具有检索增强生成(RAG)能力的LLMs来分析CoT轨迹,识别需要遗忘的段落,并用无害的占位符替换它们以保持逻辑结构。我们还引入了一种新的特征替换反向学习损失,该损失可以同时抑制生成被遗忘内容的概率,同时强化结构上有效的替换。在合成和医学数据集上的广泛实验验证了我们所提方法的期望属性。

英文摘要:

Large Reasoning Models (LRMs) generate structured chains of thought (CoTs) before producing final answers, making them especially vulnerable to knowledge leakage through intermediate reasoning steps. Yet, the memorization of sensitive information in the training data such as copyrighted and private content has led to ethical and legal concerns. To address these issues, selective forgetting (also known as machine unlearning) has emerged as a potential remedy for LRMs. However, existing unlearning methods primarily target final answers and may degrade the overall reasoning ability of LRMs after forgetting. Additionally, directly applying unlearning on the entire CoTs could degrade the general reasoning capabilities. The key challenge for LRM unlearning lies in achieving precise unlearning of targeted knowledge while preserving the integrity of general reasoning capabilities. To bridge this gap, we in this paper propose a novel LRM unlearning framework that selectively removes sensitive reasoning components while preserving general reasoning capabilities. Our approach leverages multiple LLMs with retrieval-augmented generation (RAG) to analyze CoT traces, identify forget-relevant segments, and replace them with benign placeholders that maintain logical structure. We also introduce a new feature replacement unlearning loss for LRMs, which can simultaneously suppress the probability of generating forgotten content while reinforcing structurally valid replacements. Extensive experiments on both synthetic and medical datasets verify the desired properties of our proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑