发表机构
Zhangjiang Lab of Artificial Intelligence（上海人工智能实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
Diffu-LoRA是一种参数高效的个性化扩散模型方法，通过门控低秩适配学习层间非均匀的适配能力分配，在Stable Diffusion实验中提升了主体保真度与提示对齐度。
AI 中文摘要
从少量参考图像对文本到图像的扩散模型进行个性化，需要在遵循描述新上下文的提示的同时保留主体身份。全模型微调参数密集，而低秩适配（LoRA）减少了可训练参数的数量，但未解决适配能力应如何在各层间分配的问题。我们提出Diffu-LoRA，这是一种参数高效的方法，通过门控低秩适配学习这种分配。Diffu-LoRA在Transformer块的线性层中插入可训练的低秩组件，并为每个组件分配一个可学习的门。双层优化在不同的数据分割上更新适配权重和门参数，同时渐进式剪枝移除门值最低的组件，以满足规定的秩预算。该过程在保持预训练主干冻结的同时，在各层间非均匀地分配适配能力。在Stable Diffusion上针对DreamBooth的主体及额外收集的数据集进行的实验表明，与所评估的微调基线相比，整体主体保真度和提示对齐度均有所提升。 ablation研究考察了双层优化、渐进式剪枝和适配器放置的贡献。这些结果支持学习到的秩分配作为参数高效的扩散模型个性化的实用方法。
英文摘要
Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers. We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen. Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement. These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.
