撤销重要信息：针对语言模型精确撤销的标记级归因
Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearning
AI总结：
本文提出TokenUnlearn框架，通过标记级归因实现精确撤销，结合知识意识信号和熵意识信号生成重要性评分，改进梯度信噪比，实验显示在遗忘效果和效用保持方面优于序列级基线。
摘要：
机器撤销已成为解决大型语言模型（LLMs）隐私、安全和监管问题的关键能力。现有方法在序列层面操作，对所有标记应用统一更新，尽管只有部分标记编码需移除的知识。这引入了梯度噪声，降低了效用，并导致次优遗忘。我们提出TokenUnlearn，一种标记级归因框架，识别并选择性地针对关键标记。我们的方法通过掩码结合知识意识信号，并通过熵意识信号生成重要性评分以实现精确标记选择。我们开发了两种互补策略：硬选择，仅对高重要性标记应用撤销；软加权，根据重要性评分调节梯度贡献。两者均扩展了现有方法到标记级变体。理论分析显示标记级选择提高了梯度信噪比。在TOFU和WMDP基准上，三种模型架构的实验显示在遗忘效果和效用保持方面，相对于序列级基线有持续改进。
英文摘要：
Machine unlearning has emerged as a critical capability for addressing privacy, safety, and regulatory concerns in large language models (LLMs). Existing methods operate at the sequence level, applying uniform updates across all tokens despite only a subset encoding the knowledge targeted for removal. This introduces gradient noise, degrades utility, and leads to suboptimal forgetting. We propose TokenUnlearn, a token-level attribution framework that identifies and selectively targets critical tokens. Our approach combines knowledge-aware signals via masking, and entropy-aware signals to yield importance scores for precise token selection. We develop two complementary strategies: hard selection, applying unlearning only to high-importance tokens, and soft weighting, modulating gradient contributions based on importance scores. Both extend existing methods to token-level variants. Theoretical analysis shows token-level selection improves gradient signal-to-noise ratio. Experiments on TOFU and WMDP benchmarks across three model architectures demonstrate consistent improvements over sequence-level baselines in both forgetting effectiveness and utility preservation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
