Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mecha- nism that can incorporate external constraints from diverse sources in- cluding a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent coloriza- tions on the LFW, LSUN-Church and ILSVRC-2015 datasets.
ControlCol: Controllability in Automatic Speaker Video Colorization
SFI Centre for Research Training in Artificial Intelligence（SFI人工智能研究培训中心）；Data Science Institute（数据科学研究所）；University of Galway（戈尔韦大学）；School of Engineering（工程学院）
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Nanyang Technological University（南洋理工大学）；Applied Research Center (ARC), Tencent PCG（腾讯平台与内容事业群应用研究中心）
Video colorization is a challenging and highly ill-posed problem. Although recent years have witnessed remarkable progress in single image colorization, there is relatively less research effort on video colorization and existing methods always suffer from severe flickering artifacts (temporal inconsistency) or unsatisfying colorization performance. We address this problem from a new perspective, by jointly considering colorization and temporal consistency in a unified framework. Specifically, we propose a novel temporally consistent video colorization framework (TCVC). TCVC effectively propaga…
Hong Kong University of Science and Technology（香港科技大学）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；City University of Hong Kong（香港城市大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft AI Perception and Mixed Reality（微软人工智能感知与混合现实部门）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
This paper presents the first end-to-end network for exemplar-based video colorization. The main challenge is to achieve temporal consistency while remaining faithful to the reference style. To address this issue, we introduce a recurrent framework that unifies the semantic correspondence and color propagation steps. Both steps allow a provided reference image to guide the colorization of every frame, thus reducing accumulated propagation errors. Video frames are colorized in sequence based on the colorization history, and its coherency is further enforced by the temporal consistency loss. All…
Unsupervised deep learning has recently demonstrated the promise of producing high-quality samples. While it has tremendous potential to promote the image colorization task, the performance is limited owing to the high-dimension of data manifold and model capability. This study presents a novel scheme that exploits the score-based generative model in wavelet domain to address the issues. By taking advantage of the multi-scale and multi-channel representation via wavelet transform, the proposed model learns the richer priors from stacked coarse and detailed wavelet coefficient components jointl…
Hacettepe University（哈斯特帕大学）；Ecole des Ponts（巴黎路桥学院）；Univ Gustave Eiffel（古斯塔夫·埃菲尔大学）；CNRS（法国国家科学研究中心）；Middle East Technical University（中东技术大学）
We propose a new method for producing color images from sketches. Current solutions in sketch colorization either necessitate additional user instruction or are restricted to the "paired" translation strategy. We leverage semantic image segmentation from a general-purpose panoptic segmentation network to generate an additional adversarial loss function. The proposed loss function is compatible with any GAN model. Our method is not restricted to datasets with segmentation labels and can be applied to unpaired translation tasks as well. Using qualitative, and quantitative analysis, and based on …
LatentColorization: Latent Diffusion-Based Speaker Video Colorization
SFI Centre for Research Training in Artificial Intelligence（SFI人工智能研究培训中心）；University of Galway（戈尔韦大学）；School of Computer Science（计算机科学学院）；School of Engineering（工程学院）
We present a fully automatic approach to video colorization with self-regularization and diversity. Our model contains a colorization network for video frame colorization and a refinement network for spatiotemporal color refinement. Without any labeled data, both networks can be trained with self-regularized losses defined in bilateral and temporal space. The bilateral loss enforces color consistency between neighboring pixels in a bilateral space and the temporal loss imposes constraints between corresponding pixels in two nearby frames. While video colorization is a multi-modal problem, our …
Automatic animation line art colorization is a challenging computer vision problem, since the information of the line art is highly sparse and abstracted and there exists a strict requirement for the color and style consistency between frames. Recently, a lot of Generative Adversarial Network (GAN) based image-to-image translation methods for single line art colorization have emerged. They can generate perceptually appealing results conditioned on line art images. However, these methods can not be adopted for the purpose of animation colorization because there is a lack of consideration of the…
CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility
The Chinese University of Hong Kong（香港中文大学）；International Digital Economy Academy (IDEA)（国际数字经济学院）；The University of Hong Kong（香港大学）；Tsinghua University（清华大学）
FSCM: Frequency-Enhanced Spatial-Spectral Coupled Mamba for Infrared Hyperspectral Image Colorization
School of Electronic and Optical Engineering, Nanjing University of Science and Technology（南京理工大学电子与光学工程学院）；School of Mechanical Engineering, University of Science and Technology Beijing（北京科技大学机械工程学院）；School of Instrument and Electronics, North University of China（北方大学仪器与电子学院）
Scarcity of labeled data has motivated the development of semi-supervised learning methods, which learn from large portions of unlabeled data alongside a few labeled samples. Consistency Regularization between model's predictions under different input perturbations, particularly has shown to provide state-of-the art results in a semi-supervised framework. However, most of these method have been limited to classification and segmentation applications. We propose Transformation Consistency Regularization, which delves into a more challenging setting of image-to-image translation, which remains u…
