arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.14736cs.CV
收藏

TextIR:一个简单的基于文本的可编辑图像复原框架

TextIR: A Simple Framework for Text-based Editable Image Restoration

Tsinghua University(清华大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Shanghai AI Laboratory(上海人工智能实验室);Peng Cheng Laboratory(鹏城实验室)

Yunpeng Bai, Cairong Wang, Shuzhao Xie, Chao Dong, Chun Yuan, Zhi Wang

摘要:

现有的大多数图像复原方法使用神经网络从大量数据中学习强图像级先验来估计丢失的信息。然而,这些作品仍然挣扎的情况下,图像有严重的信息赤字。引入外部先验或使用参考图像来提供信息在应用领域中也具有局限性。相比之下,文本输入更容易获得,并且提供具有更高灵活性的信息。在这项工作中,我们设计了一个有效的框架,允许用户控制退化图像的恢复过程与文字描述。我们利用CLIP的文本-图像特征兼容性来减轻文本和图像特征融合的困难。我们的框架可以用于各种图像复原任务,包括图像修补,图像超分辨率,和图像彩色化。大量的实验证明了我们的方法的有效性。

英文摘要:

Most existing image restoration methods use neural networks to learn strong image-level priors from huge data to estimate the lost information. However, these works still struggle in cases when images have severe information deficits. Introducing external priors or using reference images to provide information also have limitations in the application domain. In contrast, text input is more readily available and provides information with higher flexibility. In this work, we design an effective framework that allows the user to control the restoration process of degraded images with text descriptions. We use the text-image feature compatibility of the CLIP to alleviate the difficulty of fusing text and image features. Our framework can be used for various image restoration tasks, including image inpainting, image super-resolution, and image colorization. Extensive experiments demonstrate the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑