arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.07600cs.CV
收藏

面向基于扩散的视觉感知的隐式与显式语言引导

Implicit and Explicit Language Guidance for Diffusion-based Visual Perception

School of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Big Data and Software Engineering, Chongqing University(重庆大学大数据与软件学院)

Hefeng Wang, Jiale Cao, Jin Xie, Aiping Yang, Yanwei Pang

AI总结:

针对预训练扩散模型适配视觉感知的难题,提出含隐式、显式双语言分支的IEDP框架,训练时联合引导特征学习,推理仅用隐式分支,在语义分割和深度估计任务上性能优于基线VPD。

摘要:

文本到图像扩散模型已在条件图像合成任务中展现出强大能力。依托大规模视觉-语言预训练,扩散模型能够在不同文本提示下生成纹理丰富、结构合理的高质量图像。然而,如何将预训练扩散模型适配到视觉感知任务仍是一个待解决的开放性问题。本文提出一种面向基于扩散的感知的隐式与显式语言引导框架,命名为IEDP。该框架包含隐式语言引导分支和显式语言引导分支:隐式分支采用冻结的CLIP图像编码器,直接生成隐式文本嵌入并输入扩散模型,无需使用显式文本提示;显式分支则将对应图像的真实标签作为文本提示,为扩散模型的特征提取提供条件约束。训练阶段,两个分支共享模型权重,对扩散模型进行联合训练,从而实现隐式与显式分支对特征学习的共同引导。推理阶段仅使用隐式分支完成最终预测,无需任何真实标签。实验在语义分割和深度估计两个典型感知任务上开展,IEDP在两项任务中均取得了优异性能:语义分割任务上,IEDP在AD20K验证集上的mIoU$^\text{ss}$得分为55.9%,较基线方法VPD高出2.2%;深度估计任务上,IEDP较基线方法VPD实现了11.0%的相对提升。

英文摘要:

Text-to-image diffusion models have shown powerful ability on conditional image synthesis. With large-scale vision-language pre-training, diffusion models are able to generate high-quality images with rich texture and reasonable structure under different text prompts. However, it is an open problem to adapt the pre-trained diffusion model for visual perception. In this paper, we propose an implicit and explicit language guidance framework for diffusion-based perception, named IEDP. Our IEDP comprises an implicit language guidance branch and an explicit language guidance branch. The implicit branch employs frozen CLIP image encoder to directly generate implicit text embeddings that are fed to diffusion model, without using explicit text prompts. The explicit branch utilizes the ground-truth labels of corresponding images as text prompts to condition feature extraction of diffusion model. During training, we jointly train diffusion model by sharing the model weights of these two branches. As a result, implicit and explicit branches can jointly guide feature learning. During inference, we only employ implicit branch for final prediction, which does not require any ground-truth labels. Experiments are performed on two typical perception tasks, including semantic segmentation and depth estimation. Our IEDP achieves promising performance on both tasks. For semantic segmentation, our IEDP has the mIoU$^\text{ss}$ score of 55.9% on AD20K validation set, which outperforms the baseline method VPD by 2.2%. For depth estimation, our IEDP outperforms the baseline method VPD with a relative gain of 11.0%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑