针对视频上色中颜色不饱和与时间不一致问题，提出基于Stable Video Diffusion的生成流程，设计调色板颜色引导器提供颜色指导并增强跨序列稳定性，实验证明其能生成生动稳定颜色且超越现有方法。
摘要：
上色是一项传统的计算机视觉任务，在许多耗时任务中发挥着重要作用，例如老电影修复。现有方法存在颜色不饱和以及时间上不一致的问题。在本文中，我们提出了一种新颖的流程来克服这些挑战。我们将上色任务视为生成任务，并引入Stable Video Diffusion（SVD）作为基础模型。我们设计了一个基于调色板的颜色引导器，以协助模型生成生动且一致的颜色。调色板引入的颜色上下文不仅为颜色生成提供指导，而且通过跨多个序列的统一颜色上下文增强了生成颜色的稳定性。实验表明，所提出的方法能够为视频提供生动且稳定的颜色，超越了以前的方法。
英文摘要：
Colorization is a traditional computer vision task and it plays an important role in many time-consuming tasks, such as old film restoration. Existing methods suffer from unsaturated color and temporally inconsistency. In this paper, we propose a novel pipeline to overcome the challenges. We regard the colorization task as a generative task and introduce Stable Video Diffusion (SVD) as our base model. We design a palette-based color guider to assist the model in generating vivid and consistent colors. The color context introduced by the palette not only provides guidance for color generation, but also enhances the stability of the generated colors through a unified color context across multiple sequences. Experiments demonstrate that the proposed method can provide vivid and stable colors for videos, surpassing previous methods.
Shanghai AI Laboratory（上海人工智能实验室）；Tencent AI Lab（腾讯人工智能实验室）；CUHK（香港中文大学）；SenseTime Research（商汤科技研究院）
Multimodal ambiguity and color bleeding remain challenging in colorization. To tackle these problems, we propose a new GAN-based colorization approach PalGAN, integrated with palette estimation and chromatic attention. To circumvent the multimodality issue, we present a new colorization formulation that estimates a probabilistic palette from the input gray image first, then conducts color assignment conditioned on the palette through a generative model. Further, we handle color bleeding with chromatic attention. It studies color affinities by considering both semantic and intensity correlation…
Korea University（高丽大学）；Clova AI Research, NAVER Corp.（NAVER公司Clova AI研究中心）；Hong Kong University of Science and Technology（香港科技大学）
This paper proposes a novel approach to generate multiple color palettes that reflect the semantics of input text and then colorize a given grayscale image according to the generated color palette. In contrast to existing approaches, our model can understand rich text, whether it is a single word, a phrase, or a sentence, and generate multiple possible palettes from it. For this task, we introduce our manually curated dataset called Palette-and-Text (PAT). Our proposed model called Text2Colors consists of two conditional generative adversarial networks: the text-to-palette generation networks …
The Hong Kong University of Science and Technology（香港科技大学）
We present a novel approach to automatic image colorization by imitating the imagination process of human experts. Our imagination module is designed to generate color images that are context-correlated with black-and-white photos. Given a black-and-white image, our imagination module firstly extracts the context information, which is then used to synthesize colorful and diverse images using a conditional image synthesis network (e.g., semantic image synthesis model). We then design a colorization module to colorize the black-and-white images with the guidance of imagination for photorealistic…
Hacettepe University（哈斯特帕大学）；Middle East Technical University（中东技术大学）
We introduce a new method for generating color images from sketches or edge maps. Current methods either require some form of additional user-guidance or are limited to the "paired" translation approach. We argue that segmentation information could provide valuable guidance for sketch colorization. To this end, we propose to leverage semantic image segmentation, as provided by a general purpose panoptic segmentation network, to create an additional adversarial loss function. Our loss function can be integrated to any baseline GAN model. Our method is not limited to datasets that contain segmen…
Nanjing University of Science and Technology（南京理工大学）；Communication University of China（中国传媒大学）
How to effectively explore the colors of reference exemplars and propagate them to colorize each frame is vital for exemplar-based video colorization. In this paper, we present an effective BiSTNet to explore colors of reference exemplars and utilize them to help video colorization by a bidirectional temporal feature fusion with the guidance of semantic image prior. We first establish the semantic correspondence between each frame and the reference exemplars in deep feature space to explore color information from reference exemplars. Then, to better propagate the colors of reference exemplars …
