控制视觉-语言模型以实现多任务图像复原
Controlling Vision-Language Models for Multi-Task Image Restoration
AI总结：
本文提出 DA-CLIP，通过退化感知控制器适配固定 CLIP 编码器并以交叉注意力引导复原网络，实现多任务图像复原并提升最先进性能。
摘要：
诸如 CLIP 之类的视觉-语言模型已在多种下游任务的零样本或无标签预测中展现出巨大影响。然而，在图像复原等低层视觉任务中，由于输入受到损坏，它们的性能会急剧下降。本文提出一种退化感知视觉-语言模型（DA-CLIP），以更好地将预训练视觉-语言模型迁移到低层视觉任务，作为面向图像复原的多任务框架。具体而言，DA-CLIP 训练一个额外的控制器，使固定的 CLIP 图像编码器能够适配并预测高质量特征嵌入。通过交叉注意力将该嵌入整合到图像复原网络中，我们能够引导模型学习高保真图像重建。控制器本身还会输出与输入真实损坏相匹配的退化特征，从而自然形成一个针对不同退化类型的分类器。此外，我们构建了一个带有合成描述文本的混合退化数据集，用于 DA-CLIP 训练。我们的方法在退化特定图像复原任务和统一图像复原任务上均推进了最先进性能，展示了利用大规模预训练视觉-语言模型提示图像复原的有前景方向。代码位于 https://github.com/Algolzw/daclip-uir。
英文摘要：
Vision-language models such as CLIP have shown great impact on diverse downstream tasks for zero-shot or label-free predictions. However, when it comes to low-level vision such as image restoration their performance deteriorates dramatically due to corrupted inputs. In this paper, we present a degradation-aware vision-language model (DA-CLIP) to better transfer pretrained vision-language models to low-level vision tasks as a multi-task framework for image restoration. More specifically, DA-CLIP trains an additional controller that adapts the fixed CLIP image encoder to predict high-quality feature embeddings. By integrating the embedding into an image restoration network via cross-attention, we are able to pilot the model to learn a high-fidelity image reconstruction. The controller itself will also output a degradation feature that matches the real corruptions of the input, yielding a natural classifier for different degradation types. In addition, we construct a mixed degradation dataset with synthetic captions for DA-CLIP training. Our approach advances state-of-the-art performance on both \emph{degradation-specific} and \emph{unified} image restoration tasks, showing a promising direction of prompting image restoration with large-scale pretrained vision-language models. Our code is available at https://github.com/Algolzw/daclip-uir.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
