arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12004cs.CV
收藏

面向文本到图像扩散模型鲁棒定制的直接一致性优化

Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models

KAIST(韩国科学技术院);Google Research(谷歌研究院);Meta Reality Labs(元宇宙现实实验室)

Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, Jinwoo Shin

AI总结:

针对文本到图像扩散模型微调后鲁棒性差、难以跨模型概念组合的问题,提出直接一致性优化微调目标,通过控制与预训练模型的偏差保留预训练知识,在一致性与对齐度的帕累托前沿上优于基线,且支持模型无干扰合并。

摘要:

文本到图像(T2I)扩散模型在少量个人图像上微调后,能够生成高度一致的视觉内容。但这类微调后的模型鲁棒性不足,往往无法与预训练模型或其他微调模型的概念进行组合。为解决这一问题,我们提出一种新型微调目标,名为Direct Consistency Optimization(直接一致性优化),它通过控制微调模型与预训练模型之间的偏差,在微调过程中保留预训练知识。我们在主体定制和风格定制任务上开展了大量实验,结果表明,该方法在主体(或风格)一致性与图文对齐度之间的帕累托前沿上优于所有现有基线;它不仅在图文对齐度上优于常规微调目标,而且相比使用额外先验数据集进行微调的方法,对参考图像的保真度更高。更重要的是,采用我们方法微调的模型可以无干扰地合并,通过组合分别定制的主体模型和风格模型,就能生成具有定制风格的定制主体。值得注意的是,我们的方法在提示保真度和主体保真度上,均优于针对常规微调模型合并进行后优化的方法。

英文摘要:

Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, can generate visuals with a high degree of consistency. However, such fine-tuned models are not robust; they often fail to compose with concepts of pretrained model or other fine-tuned models. To address this, we propose a novel fine-tuning objective, dubbed Direct Consistency Optimization, which controls the deviation between fine-tuning and pretrained models to retain the pretrained knowledge during fine-tuning. Through extensive experiments on subject and style customization, we demonstrate that our method positions itself on a superior Pareto frontier between subject (or style) consistency and image-text alignment over all previous baselines; it not only outperforms regular fine-tuning objective in image-text alignment, but also shows higher fidelity to the reference images than the method that fine-tunes with additional prior dataset. More importantly, the models fine-tuned with our method can be merged without interference, allowing us to generate custom subjects in a custom style by composing separately customized subject and style models. Notably, we show that our approach achieves better prompt fidelity and subject fidelity than those post-optimized for merging regular fine-tuned models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑