arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.03213cs.LG
收藏

基于批评者的文本到图像扩散强化学习

Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusion

SoftServe, Sadova St, 2D, Lviv, Lviv Oblast Ukraine(SoftServe);Ukrainian Catholic University(乌克兰天主教大学)

Mykola Vysotskyi, Zahar Kohut, Mariia Shpir, Taras Rumezhak, Volodymyr Karpiv

AI总结:

本文提出了一种基于批评者的强化学习框架,用于文本到图像扩散模型的无学习,通过时间步感知的批评者和噪声时间步奖励提升稳定性和效果。

摘要:

机器无学习在文本到图像扩散模型中旨在在去除目标概念的同时保持整体效用。先前的扩散无学习方法通常依赖于监督的权重编辑或全局惩罚;强化学习(RL)方法虽然灵活,但往往优化稀疏的轨迹末端奖励,导致高方差更新和弱信用分配。我们提出了一种通用的RL框架用于扩散无学习,将去噪视为一个顺序决策过程,并引入了时间步感知的批评者和带有噪声时间步奖励的奖励。具体来说,我们训练了一个基于CLIP的奖励预测器在噪声潜在空间上,并利用其每步信号来计算策略梯度更新反向扩散内核的优势估计。我们的算法易于实现,支持非策略重用,并可插入标准的文本到图像骨干网络。在多个概念上,该方法在遗忘效果上优于或可与强基线相媲美,同时保持图像质量和良性提示保真度;消融研究显示(i)每步批评者和(ii)噪声条件化奖励对稳定性和有效性至关重要。我们释放代码和评估脚本以促进可重复性和未来基于RL的扩散无学习研究。

英文摘要:

Machine unlearning in text-to-image diffusion models aims to remove targeted concepts while preserving overall utility. Prior diffusion unlearning methods typically rely on supervised weight edits or global penalties; reinforcement-learning (RL) approaches, while flexible, often optimize sparse end-of-trajectory rewards, yielding high-variance updates and weak credit assignment. We present a general RL framework for diffusion unlearning that treats denoising as a sequential decision process and introduces a timestep-aware critic with noisy-step rewards. Concretely, we train a CLIP-based reward predictor on noisy latents and use its per-step signal to compute advantage estimates for policy-gradient updates of the reverse diffusion kernel. Our algorithm is simple to implement, supports off-policy reuse, and plugs into standard text-to-image backbones. Across multiple concepts, the method achieves better or comparable forgetting to strong baselines while maintaining image quality and benign prompt fidelity; ablations show that (i) per-step critics and (ii) noisy-conditioned rewards are key to stability and effectiveness. We release code and evaluation scripts to facilitate reproducibility and future research on RL-based diffusion unlearning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑