arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.14733cs.CV
收藏

利用元提示的扩散模型进行视觉感知

Harnessing Diffusion Models for Visual Perception with Meta Prompts

Fudan University(复旦大学);ByteDance Inc(字节跳动公司)

Qiang Wan, Zilong Huang, Bingyi Kang, Jiashi Feng, Li Zhang

AI总结:

本文提出利用预训练扩散模型,通过引入可学习的元提示和循环细化训练策略,有效提取视觉特征,在深度估计和语义分割等任务上取得领先性能。

摘要:

视觉模型的生成式预训练问题一直是一个长期存在的难题。目前,文本到图像(T2I)扩散模型在生成与文本输入匹配的高清图像方面表现出卓越的能力,这一成就得益于其在大规模图像-文本对上的预训练。这引发了一个自然的疑问:能否利用扩散模型来处理视觉感知任务?在本文中,我们提出了一种简单而有效的方案,利用扩散模型完成视觉感知任务。我们的关键见解是在预训练的扩散模型中引入可学习的嵌入(元提示),以提取适合感知的特征。元提示的作用有两个方面。首先,作为T2I模型中文本嵌入的直接替代,它可以在特征提取过程中激活与任务相关的特征。其次,它被用于重新排列提取的特征,以确保模型专注于当前任务最相关的特征。此外,我们设计了一种循环细化训练策略,充分利用扩散模型的特性,从而产生更强的视觉特征。跨多个基准的大量实验验证了我们方法的有效性。我们的方法在NYU深度V2和KITTI上的深度估计任务以及CityScapes上的语义分割任务中取得了新的性能记录。同时,所提出的方法在ADE20K上的语义分割和COCO数据集上的姿态估计任务中获得了与当前最先进技术相当的结果,进一步展示了其鲁棒性和通用性。

英文摘要:

The issue of generative pretraining for vision models has persisted as a long-standing conundrum. At present, the text-to-image (T2I) diffusion model demonstrates remarkable proficiency in generating high-definition images matching textual inputs, a feat made possible through its pre-training on large-scale image-text pairs. This leads to a natural inquiry: can diffusion models be utilized to tackle visual perception tasks? In this paper, we propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. Our key insight is to introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. The effect of meta prompts are two-fold. First, as a direct replacement of the text embeddings in the T2I models, it can activate task-relevant features during feature extraction. Second, it will be used to re-arrange the extracted features to ensures that the model focuses on the most pertinent features for the task on hand. Additionally, we design a recurrent refinement training strategy that fully leverages the property of diffusion models, thereby yielding stronger visual features. Extensive experiments across various benchmarks validate the effectiveness of our approach. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes. Concurrently, the proposed method attains results comparable to the current state-of-the-art in semantic segmentation on ADE20K and pose estimation on COCO datasets, further exemplifying its robustness and versatility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑