arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.09980cs.LGcs.AIcs.CL
收藏

通过推理实现可解释的LLM反学习

Explainable LLM Unlearning Through Reasoning

Junfeng Liao, Qizhou Wang, Shanshan Ye, Xin Yu, Ling Chen, Zhen Fang

AI总结:

本文提出基于推理的反学习方法TRU,通过引入推理引导的反学习目标,实现更可靠且可解释的LLM反学习,同时保持模型通用能力。

摘要:

LLM反学习对于缓解预训练大语言模型(LLMs)中的安全、版权和隐私问题至关重要。与偏好对齐相比,它提供了一种更明确的方法,通过特定的反学习数据集去除不受欢迎的知识。在先前的工作中,梯度上升(GA)及其变种在实现反学习方面显示出潜力,但其非目标性导致了通用能力的意外退化、知识去除不完全以及生成不连贯的响应等问题。我们认为这些问题的根源在于缺乏对模型应如何反学习的明确指导。为填补这一空白,我们引入了一个新的反学习目标,即基于推理的反学习目标,它既满足指定的反学习范围,又满足指定反学习后的响应。在此基础上,我们提出了目标推理反学习(TRU),它利用基于推理的反学习目标作为指导。我们通过交叉熵监督损失结合GA-based损失来利用该目标,使模型能够学习推理能力以精确去除知识,同时保持无关能力。我们评估了TRU在多个基准和LLM基础架构上的表现,发现其在保持通用能力的同时实现了更可靠的反学习。此外,TRU在多样化的攻击场景下表现出更强的鲁棒性,这源于通过基于推理的目标学习的推理能力。总体而言,我们的研究确立了基于推理的反学习作为可靠且可解释的LLM反学习实用范式。

英文摘要:

LLM unlearning is essential for mitigating safety, copyright, and privacy concerns in pre-trained large language models (LLMs). Compared to preference alignment, it offers a more explicit way by removing undesirable knowledge characterized by specific unlearning datasets. In previous works, gradient ascent (GA) and its variants have shown promise for implementing unlearning, yet their untargeted nature results in unintended degradation of general capabilities, incomplete removal of knowledge, and the generation of incoherent responses, among many others. We argue that these issues stem from the absence of explicit guidance on what and how models should unlearn. To fill this gap, we introduce a novel unlearning target, reasoning-based unlearning target, which satisfies both the specified unlearning scope and the specified post-unlearning response. Building on this, we propose targeted reasoning unlearning (TRU), which leverages reasoning-based unlearning target as guidance. We employ the target using a cross-entropy supervised loss combined with a GA-based loss, enabling the model to learn reasoning ability for precise knowledge removal while preserving unrelated abilities. We evaluate TRU against strong baselines across multiple benchmarks and LLM backbones, and find that it achieves more reliable unlearning while preserving general capabilities. Moreover, TRU exhibits superior robustness under diverse attack scenarios, stemming from the reasoning ability learned through reasoning-based targets. Overall, our study establishes reasoning-augmented unlearning as a practical paradigm for reliable and explainable LLM unlearning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑