arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.11223cs.CVcs.GR
收藏

UniColor:一个基于Transformer的多模态彩色化统一框架

UniColor: A Unified Framework for Multi-Modal Colorization with Transformer

City University of Hong Kong(香港城市大学);University of Bath(巴斯大学)

Zhitong Huang, Nanxuan Zhao, Jing Liao

摘要:

我们提出了第一个统一的框架UniColor,以支持多种形式的着色,包括无条件和有条件的,如中风,样本,文本,甚至它们的混合。我们不是为每种类型的条件学习一个单独的模型,而是引入一个两阶段的着色框架,将各种条件合并到一个模型中。在第一阶段中,多模态条件被转换为提示点的共同表示。特别是,我们提出了一种新的基于CLIP的方法来转换文本的提示点。在第二阶段,我们提出了一个由Chroma-VQGAN和Hybrid-Transformer组成的基于Transformer的网络,以生成以提示点为条件的多样化和高质量的着色结果。定性和定量的比较表明,我们的方法优于国家的最先进的方法,在每一个控制模式,并进一步实现多模态彩色化,这是不可行的。此外,我们还设计了一个交互式界面,展示了我们的统一框架在实际使用中的有效性,包括自动着色,混合控制着色,局部着色和迭代颜色编辑。我们的代码和模型可以在https://luckyhzt.github.io/unicolor上找到。

英文摘要:

We propose the first unified framework UniColor to support colorization in multiple modalities, including both unconditional and conditional ones, such as stroke, exemplar, text, and even a mix of them. Rather than learning a separate model for each type of condition, we introduce a two-stage colorization framework for incorporating various conditions into a single model. In the first stage, multi-modal conditions are converted into a common representation of hint points. Particularly, we propose a novel CLIP-based method to convert the text to hint points. In the second stage, we propose a Transformer-based network composed of Chroma-VQGAN and Hybrid-Transformer to generate diverse and high-quality colorization results conditioned on hint points. Both qualitative and quantitative comparisons demonstrate that our method outperforms state-of-the-art methods in every control modality and further enables multi-modal colorization that was not feasible before. Moreover, we design an interactive interface showing the effectiveness of our unified framework in practical usage, including automatic colorization, hybrid-control colorization, local recolorization, and iterative color editing. Our code and models are available at https://luckyhzt.github.io/unicolor.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑