arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.02153cs.CV
收藏

释放视觉感知的文本到图像扩散模型

Unleashing Text-to-Image Diffusion Models for Visual Perception

Tsinghua University(清华大学);University of Washington(华盛顿大学)

Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, Jiwen Lu

摘要:

扩散模型已成为生成模型的新趋势,并显示出强大的条件综合能力。其中,在大规模图像-文本对上预训练的文本到图像扩散模型通过可定制的提示高度可控。与关注低层属性和细节的无条件生成模型不同,文本到图像扩散模型由于视觉语言预训练而包含更多的高级知识。在本文中,我们提出了VPD(视觉感知与预训练的扩散模型),一个新的框架,利用语义信息的预训练的文本到图像的扩散模型在视觉感知任务。我们没有在基于扩散的管道中使用预训练的去噪自动编码器,而是简单地将其用作主干,旨在研究如何充分利用所学习的知识。具体来说,我们使用适当的文本输入提示去噪解码器,并使用适配器优化文本特征,从而更好地对齐预训练阶段,并使视觉内容与文本提示交互。我们还建议利用视觉特征和文本特征之间的交叉注意图来提供明确的指导。与其他预训练方法相比,我们表明,视觉语言预训练的扩散模型可以更快地适应下游的视觉感知任务,使用建议的VPD。通过对语义分割、参考图像分割和深度估计的实验,验证了该方法的有效性。值得注意的是,VPD在NYUv 2深度估计上达到0.254 RMSE,在RefCOCO-val参考图像分割上达到73.3% oIoU,在这两个基准上建立了新的记录。代码可在https://github.com/wl-zhao/VPD上获得

英文摘要:

Diffusion models (DMs) have become the new trend of generative models and have demonstrated a powerful ability of conditional synthesis. Among those, text-to-image diffusion models pre-trained on large-scale image-text pairs are highly controllable by customizable prompts. Unlike the unconditional generative models that focus on low-level attributes and details, text-to-image diffusion models contain more high-level knowledge thanks to the vision-language pre-training. In this paper, we propose VPD (Visual Perception with a pre-trained Diffusion model), a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. Instead of using the pre-trained denoising autoencoder in a diffusion-based pipeline, we simply use it as a backbone and aim to study how to take full advantage of the learned knowledge. Specifically, we prompt the denoising decoder with proper textual inputs and refine the text features with an adapter, leading to a better alignment to the pre-trained stage and making the visual contents interact with the text prompts. We also propose to utilize the cross-attention maps between the visual features and the text features to provide explicit guidance. Compared with other pre-training methods, we show that vision-language pre-trained diffusion models can be faster adapted to downstream visual perception tasks using the proposed VPD. Extensive experiments on semantic segmentation, referring image segmentation and depth estimation demonstrates the effectiveness of our method. Notably, VPD attains 0.254 RMSE on NYUv2 depth estimation and 73.3% oIoU on RefCOCO-val referring image segmentation, establishing new records on these two benchmarks. Code is available at https://github.com/wl-zhao/VPD

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑