arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.03439cs.CV
收藏

CleanDIFT: 没有噪声的扩散特征

CleanDIFT: Diffusion Features without Noise

Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, Björn Ommer

AI总结:

CleanDIFT通过无监督微调方法,实现无噪声的高质量扩散特征,显著提升多种任务的性能。

摘要:

大规模预训练扩散模型内部的特征最近已被确立为广泛下游任务的强大语义描述符。使用这些特征的工作通常需要在将图像输入模型之前添加噪声,以获得语义特征,因为模型在给定几乎不含噪声的图像时不会提供最有用的特征。我们证明噪声对这些特征的有用性有关键影响,这种影响无法通过使用不同随机噪声的集成来弥补。我们通过引入一种轻量级、无监督的微调方法来解决这个问题,使扩散骨干网络能够提供高质量、无噪声的语义特征。我们证明这些特征在各种提取设置和下游任务中普遍优于之前的扩散特征,其性能优于甚至比基于集成的方法更高效。

英文摘要:

Internal features from large-scale pre-trained diffusion models have recently been established as powerful semantic descriptors for a wide range of downstream tasks. Works that use these features generally need to add noise to images before passing them through the model to obtain the semantic features, as the models do not offer the most useful features when given images with little to no noise. We show that this noise has a critical impact on the usefulness of these features that cannot be remedied by ensembling with different random noises. We address this issue by introducing a lightweight, unsupervised fine-tuning method that enables diffusion backbones to provide high-quality, noise-free semantic features. We show that these features readily outperform previous diffusion features by a wide margin in a wide variety of extraction setups and downstream tasks, offering better performance than even ensemble-based methods at a fraction of the cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑