发表机构
University of Science and Technology of China（中国科学技术大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对扩散模型微调中提升特定目标生成与保留预训练能力的权衡问题，提出源先验驱动的选择性适应方法，通过关键观察学习静态掩码并构建更新策略，实验证明比基线方法在适应 - 保留权衡上表现更好。
AI 中文摘要
微调大型扩散模型以适应新领域或风格需要权衡：提升特定目标生成能力往往会降低预训练模型的广泛生成能力。现有全量和参数高效微调方法通常只是隐含地处理这种权衡。本文提出一种新颖的源先验驱动的选择性适应方法来高效微调扩散模型，实现良好权衡。该方法基于两个关键观察：预训练参数间一般生成能力损失高度不一致；对模型一般生成能力影响相对较小的参数在各层和参数类型间结构上不一致。基于此，先学习静态掩码明确识别更适合下游适应的参数，再为选定子集构建结构化更新策略。实验表明，该方法比现有强基线实现了更好的适应 - 保留权衡。
英文摘要
Fine-tuning large diffusion models for new domains or styles involves a trade-off: improving target-specific generation often degrades the pretrained model's broad generative capability. Existing full and parameter-efficient fine-tuning methods typically handle this trade-off only implicitly. In this work, we propose a novel source-prior-driven selective adaptation method to efficiently fine-tune diffusion models, achieving a favorable trade-off. Our method relies on two key observations: (1) the loss of general generative capability is highly inconsistent across pretrained parameters, and (2) parameters that have a relatively small impact on the model's general generative capability remain structurally inconsistent across layers and parameter types. Motivated by these observations, we first learn a static mask to explicitly identify parameters better suited for downstream adaptation, and then construct structured update strategies for the selected subset. Experiments show that our method achieves a better adaptation-retention trade-off than existing strong baselines.
