发表机构
AI Foundation lab; MIRAI; Applied AI Institute; BRAIn Lab; AXXX（AI 基础实验室; MIRAI; 应用人工智能研究院; BRAIn 实验室; AXXX）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出逆蒸馏反学习（IDU）统一框架，同时蒸馏多步匹配模型为一步生成器并抑制遗忘集输出，仅需遗忘数据，实验表明能降低遗忘类生成频率并保持质量。
AI 中文摘要
多步匹配模型（包括流模型和扩散模型）能够生成高质量的输出，但会带来高昂的推理成本，并且可能重现其训练数据集中不期望的组成部分。我们提出了逆蒸馏反学习（IDU），这是一个统一框架，它同时将教师多步匹配模型蒸馏为高效的一步学生生成器，并抑制对应于指定训练子集的输出。我们首先将蒸馏表述为关于数据分布的最小-最大目标，然后将该分布表示为遗忘集和生成分布的混合。这使我们能够将该混合与教师的训练分布进行比较，并在最优解处仅恢复保留的数据。我们的方法仅需要预训练的全数据教师和来自遗忘集的数据，无需访问保留的训练样本、额外的特征提取器或分类器。在流匹配和基于分数的扩散设置下，对MNIST和CIFAR-10数据集的大量实验表明，IDU显著降低了被遗忘类别的生成频率，同时保持了保留类别的生成质量。据我们所知，IDU是首个在无条件流匹配和基于分数的模型中同时进行反学习和蒸馏的统一框架。
英文摘要
Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.
