DiffSim:驯服扩散模型以评估视觉相似性
DiffSim: Taming Diffusion Models for Evaluating Visual Similarity
AI总结:
本文提出 DiffSim 方法,利用预训练扩散模型去噪 U-Net 注意力层的特征对齐来评估视觉相似性,克服了传统指标无法兼顾外观细节与语义内容的局限,并引入 Sref 和 IP 基准,在多基准上实现了与人类偏好高度一致的最先进性能。
摘要:
扩散模型从根本上改变了生成模型领域,使得评估定制化模型输出与参考输入之间的相似性变得至关重要。然而,传统的感知相似性指标主要在像素和图像块级别进行操作,比较低级颜色和纹理,但未能捕捉图像布局、物体姿态和语义内容方面的中级相似性与差异。基于对比学习的 CLIP 和基于自监督学习的 DINO 常被用于测量语义相似性,但它们高度压缩图像特征,无法充分评估外观细节。本文首次发现预训练扩散模型可用于测量视觉相似性,并引入了 DiffSim 方法,解决了传统指标在捕捉定制生成任务中感知一致性方面的局限性。通过对齐去噪 U-Net 注意力层中的特征,DiffSim 评估外观和风格相似性,展现出与人类视觉偏好更优的对齐效果。此外,我们引入了 Sref 和 IP 基准,分别用于在风格和实例层面评估视觉相似性。跨多个基准的综合评估表明,DiffSim 实现了最先进的性能,为测量生成模型中的视觉连贯性提供了稳健的工具。
英文摘要:
Diffusion models have fundamentally transformed the field of generative models, making the assessment of similarity between customized model outputs and reference inputs critically important. However, traditional perceptual similarity metrics operate primarily at the pixel and patch levels, comparing low-level colors and textures but failing to capture mid-level similarities and differences in image layout, object pose, and semantic content. Contrastive learning-based CLIP and self-supervised learning-based DINO are often used to measure semantic similarity, but they highly compress image features, inadequately assessing appearance details. This paper is the first to discover that pretrained diffusion models can be utilized for measuring visual similarity and introduces the DiffSim method, addressing the limitations of traditional metrics in capturing perceptual consistency in custom generation tasks. By aligning features in the attention layers of the denoising U-Net, DiffSim evaluates both appearance and style similarity, showing superior alignment with human visual preferences. Additionally, we introduce the Sref and IP benchmarks to evaluate visual similarity at the level of style and instance, respectively. Comprehensive evaluations across multiple benchmarks demonstrate that DiffSim achieves state-of-the-art performance, providing a robust tool for measuring visual coherence in generative models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。