Deep convolutional networks have attracted great attention in image restoration and enhancement. Generally, restoration quality has been improved by building more and more convolutional block. However, these methods mostly learn a specific model to handle all images and ignore difficulty diversity. In other words, an area in the image with high frequency tend to lose more information during compressing while an area with low frequency tends to lose less. In this article, we adrress the efficiency issue in image SR by incorporating a patch-wise rolling network(PRN) to content-adaptively recover images according to difficulty levels. In contrast to existing studies that ignore difficulty diversity, we adopt different stage of a neural network to perform image restoration. In addition, we propose a rolling strategy that utilizes the parameters of each stage more flexible. Extensive experiments demonstrate that our model not only shows a significant acceleration but also maintain state-of-the-art performance.
Personalized and content-adaptive image enhancement can find many applications in the age of social media and mobile computing. This paper presents a relative-learning-based approach, which, unlike previous methods, does not require matching original and enhanced images for training. This allows the use of massive online photo collections to train a ranking model for improved enhancement. We first propose a multi-level ranking model, which is learned from only relatively-labeled inputs that are automatically crawled. Then we design a novel parameter sampling scheme under this model to generate…
Beihang University（北京航空航天大学）；Beijing Institute of Technology（北京理工大学）；The University of Hong Kong（香港大学）
In recent years, neural image compression (NIC) algorithms have shown powerful coding performance. However, most of them are not adaptive to the image content. Although several content adaptive methods have been proposed by updating the encoder-side components, the adaptability of both latents and the decoder is not well exploited. In this work, we propose a new NIC framework that improves the content adaptability on both latents and the decoder. Specifically, to remove redundancy in the latents, our content adaptive channel dropping (CACD) method automatically selects the optimal quality leve…
The field of neural image compression has witnessed exciting progress as recently proposed architectures already surpass the established transform coding based approaches. While, so far, research has mainly focused on architecture and model improvements, in this work we explore content adaptive optimization. To this end, we introduce an iterative procedure which adapts the latent representation to the specific content we wish to compress while keeping the parameters of the network and the predictive model fixed. Our experiments show that this allows for an overall increase in rate-distortion p…
University of Electronic Science and Technology of China（电子科技大学）；Megvii Technology（旷视科技）
Image alignment by mesh warps, such as meshflow, is a fundamental task which has been widely applied in various vision applications(e.g., multi-frame HDR/denoising, video stabilization). Traditional mesh warp methods detect and match image features, where the quality of alignment highly depends on the quality of image features. However, the image features are not robust in occurrence of low-texture and low-light scenes. Deep homography methods, on the other hand, are free from such problem by learning deep features for robust performance. However, a homography is limited to plane motions. In t…
Deep convolutional neural network based image super-resolution (SR) models have shown superior performance in recovering the underlying high resolution (HR) images from low resolution (LR) images obtained from the predefined downscaling methods. In this paper we propose a learned image downscaling method based on content adaptive resampler (CAR) with consideration on the upscaling process. The proposed resampler network generates content adaptive image resampling kernels that are applied to the original HR input to generate pixels on the downscaled image. Moreover, a differentiable upscaling (…
The University of Tokyo（东京大学）；Hitachi, Ltd.（日立有限公司）
Deep image compression performs better than conventional codecs, such as JPEG, on natural images. However, deep image compression is learning-based and encounters a problem: the compression performance deteriorates significantly for out-of-domain images. In this study, we highlight this problem and address a novel task: universal deep image compression. This task aims to compress images belonging to arbitrary domains, such as natural images, line drawings, and comics. To address this problem, we propose a content-adaptive optimization framework; this framework uses a pre-trained compression mo…
Video compression is a critical component of Internet video delivery. Recent work has shown that deep learning techniques can rival or outperform human-designed algorithms, but these methods are significantly less compute and power-efficient than existing codecs. This paper presents a new approach that augments existing codecs with a small, content-adaptive super-resolution model that significantly boosts video quality. Our method, SRVC, encodes video into two bitstreams: (i) a content stream, produced by compressing downsampled low-resolution video with the existing codec, (ii) a model stream…
Technische Universität Dresden（德累斯顿工业大学）；Center for Interdisciplinary Digital Sciences (CIDS)（跨学科数字科学中心）；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig（德累斯顿/莱可扩展数据分析与人工智能中心）
Nokia Technologies（诺基亚科技）；Tampere University（坦佩雷大学）
Today, according to the Cisco Annual Internet Report (2018-2023), the fastest-growing category of Internet traffic is machine-to-machine communication. In particular, machine-to-machine communication of images and videos represents a new challenge and opens up new perspectives in the context of data compression. One possible solution approach consists of adapting current human-targeted image and video coding standards to the use case of machine consumption. Another approach consists of developing completely new compression paradigms and architectures for machine-to-machine communications. In t…
Ultra high resolution (UHR) images are almost always downsampled to fit small displays of mobile end devices and upsampled to its original resolution when exhibited on very high-resolution displays. This observation motivates us on jointly optimizing operation pairs of downsampling and upsampling that are spatially adaptive to image contents for maximal rate-distortion performance. In this paper, we propose an adaptive downsampled dual-layer (ADDL) image compression system. In the ADDL compression system, an image is reduced in resolution by learned content-adaptive downsampling kernels and co…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
This paper proposes an efficient content adaptive screen image scaling scheme for the real-time screen applications like remote desktop and screen sharing. In the proposed screen scaling scheme, a screen content classification step is first introduced to classify the screen image into text and pictorial regions. Afterward, we propose an adaptive shift linear interpolation algorithm to predict the new pixel values with the shift offset adapted to the content type of each pixel. The shift offset for each screen content type is offline optimized by minimizing the theoretical interpolation error b…
Source camera identification is still a hard task in forensics community, especially for the case of the small query image size. In this paper, we propose a solution to identify the source camera of the small-size images: content-adaptive fusion network. In order to learn better feature representation from the input data, content-adaptive convolutional neural networks(CA-CNN) are constructed. We add a convolutional layer in preprocessing stage. Moreover, with the purpose of capturing more comprehensive information, we parallel three CA-CNNs: CA3-CNN, CA5-CNN, CA7-CNN to get the content-adaptiv…
Content-Adaptive Image Retouching Guided by Attribute-Based Text Representation
School of Computer Science and Technology/School of Artificial Intelligence, China University of Mining and Technology（计算机科学与技术学院/人工智能学院，中国矿业大学）；Mine Digitization Engineering Research Center of the Ministry of Education, China University of Mining and Technology（教育部矿山数字化工程研究中心，中国矿业大学）；School of Artificial Intelligence, Xidian University（人工智能学院，西安电子科技大学）；School of Electrical Engineering and Computer Science, University of Ottawa（电气与计算机工程学院，多伦多大学）
Transformer-Guided Content-Adaptive Graph Learning for Hyperspectral Unmixing
School of Automation Engineering, Shanghai University of Electric Power（上海电力大学自动化工程学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
Compression and reconstruction of visual data have been widely studied in the computer vision community, even before the popularization of deep learning. More recently, some have used deep learning to improve or refine existing pipelines, while others have proposed end-to-end approaches, including autoencoders and implicit neural representations, such as SIREN and NeRV. In this work, we propose Neural Visual Representation with Content-adaptive Embedding (CNeRV), which combines the generalizability of autoencoders with the simplicity and compactness of implicit representation. We introduce a n…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
Objects at different spatial positions in an image exhibit different scales. Adaptive receptive fields are expected to capture suitable ranges of context for accurate pixel level semantic prediction. Recently, atrous convolution with different dilation rates has been used to generate features of multi-scales through several branches which are then fused for prediction. However, there is a lack of explicit interaction among the branches of different scales to adaptively make full use of the contexts. In this paper, we propose a Content-Adaptive Scale Interaction Network (CASINet) to exploit the…
CANeRV: Content Adaptive Neural Representation for Video Compression
School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Bytedance Inc.（字节跳动公司）；School of Computer Science, Peking University（北京大学计算机学院）
In collaborative intelligence, an artificial intelligence (AI) model is typically split between an edge device and the cloud. Feature tensors produced by the edge sub-model are sent to the cloud via an imperfect communication channel. At the cloud side, parts of the feature tensor may be missing due to packet loss. In this paper we propose a method called Content-Adaptive Linear Tensor Completion (CALTeC) to recover the missing feature data. The proposed method is fast, data-adaptive, does not require pre-training, and produces better results than existing methods for tensor data recovery in c…
University College London（伦敦大学学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Imperial College London（帝国理工学院）
State-of-the-art methods for computer vision rely heavily on the translation equivariance and spatial sharing properties of convolutional layers without explicitly taking into consideration the input content. Modern techniques employ deep sophisticated architectures in order to circumvent this issue. In this work, we propose a Pixel Adaptive Filtering Unit (PAFU) which introduces a differentiable kernel selection mechanism paired with a discrete, learnable and decorrelated group of kernels to allow for content-based spatial adaptation. First, we demonstrate the applicability of the technique i…
National Yang Ming Chiao Tung University（国立阳明交通大学）
This paper introduces a learned hierarchical B-frame coding scheme in response to the Grand Challenge on Neural Network-based Video Coding at ISCAS 2023. We address specifically three issues, including (1) B-frame coding, (2) YUV 4:2:0 coding, and (3) content-adaptive variable-rate coding with only one single model. Most learned video codecs operate internally in the RGB domain for P-frame coding. B-frame coding for YUV 4:2:0 content is largely under-explored. In addition, while there have been prior works on variable-rate coding with conditional convolution, most of them fail to consider the …
University of Michigan（密歇根大学）；Portland State University（波特兰州立大学）
Preprocessing defenses such as pixel discretization are appealing to remove adversarial attacks due to their simplicity. However, they have been shown to be ineffective except on simple datasets like MNIST. We hypothesize that existing discretization approaches failed because using a fixed codebook for the entire dataset limits their ability to balance image representation and codeword separability. We first formally prove that adaptive codebooks can provide stronger robustness guarantees than fixed codebooks as a preprocessing defense on some datasets. Based on that insight, we propose a cont…
National Yang Ming Chiao Tung University（国立阳明交通大学）
This paper introduces an online motion rate adaptation scheme for learned video compression, with the aim of achieving content-adaptive coding on individual test sequences to mitigate the domain gap between training and test data. It features a patch-level bit allocation map, termed the $α$-map, to trade off between the bit rates for motion and inter-frame coding in a spatially-adaptive manner. We optimize the $α$-map through an online back-propagation scheme at inference time. Moreover, we incorporate a look-ahead mechanism to consider its impact on future frames. Extensive experimental resul…
Simon Fraser University（西蒙菲莎大学）；Adobe Research（Adobe研究院）
Neural networks have shown great abilities in estimating depth from a single image. However, the inferred depth maps are well below one-megapixel resolution and often lack fine-grained details, which limits their practicality. Our method builds on our analysis on how the input resolution and the scene structure affects depth estimation performance. We demonstrate that there is a trade-off between a consistent scene structure and the high-frequency details, and merge low- and high-resolution estimations to take advantage of this duality using a simple depth merging network. We present a double …
Convolutional neural network (CNN) is widely used in computer vision applications. In the networks that deal with images, CNNs are the most time-consuming layer of the networks. Usually, the solution to address the computation cost is to decrease the number of trainable parameters. This solution usually comes with the cost of dropping the accuracy. Another problem with this technique is that usually the cost of memory access is not taken into account which results in insignificant speedup gain. The number of operations and memory access in a standard convolution layer is independent of the inp…
Shanghai Jiao Tong University（上海交通大学）；The University of Sydney（悉尼大学）
Recently, learning based video compression methods attract increasing attention. However, the previous works suffer from error propagation due to the accumulation of reconstructed error in inter predictive coding. Meanwhile, the previous learning based video codecs are also not adaptive to different video contents. To address these two problems, we propose a content adaptive and error propagation aware video compression system. Specifically, our method employs a joint training strategy by considering the compression performance of multiple consecutive frames instead of a single frame. Based on…
Only-Style: Stylistic Consistency in Image Generation without Content Leakage
School of Electrical & Computer Engineering, National Technical University of Athens（电气与计算机工程学院，国家技术大学雅典分校）；Robotics Institute, Athena Research Center（机器人研究所，雅典研究中心）；HERON - Center of Excellence in Robotics（HERON-机器人卓越中心）
Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence（伯努利数学、计算机科学与人工智能研究所）；University of Groningen（格罗宁根大学）
There are two types of information in each handwritten word image: explicit information which can be easily read or derived directly, such as lexical content or word length, and implicit attributes such as the author's identity. Whether features learned by a neural network for one task can be used for another task remains an open question. In this paper, we present a deep adaptive learning method for writer identification based on single-word images using multi-task learning. An auxiliary task is added to the training process to enforce the emergence of reusable features. Our proposed method t…
ADCD-Net: Robust Document Image Forgery Localization via Adaptive DCT Feature and Hierarchical Content Disentanglement
State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau（澳门大学物联网智能城市国家重点实验室，计算机与信息科学系）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）
Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time …
The domain gap caused mainly by variable medical image quality renders a major obstacle on the path between training a segmentation model in the lab and applying the trained model to unseen clinical data. To address this issue, domain generalization methods have been proposed, which however usually use static convolutions and are less flexible. In this paper, we propose a multi-source domain generalization model based on the domain and content adaptive convolution (DCAC) for the segmentation of medical images across different modalities. Specifically, we design the domain adaptive convolution …
Tuning-Free Adaptive Style Incorporation for Structure-Consistent Text-Driven Style Transfer
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Sichuan Provincial Key Laboratory for Human Disease Gene Study and the Center for Medical Genetics, Department of Laboratory Medicine, Sichuan Academy of Medical Sciences and Sichuan Provincial People’s Hospital, UESTC（四川省人类疾病基因研究重点实验室和医学遗传中心，四川省医学研究院及四川省人民医院，电子科技大学）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（深圳先进研究院，电子科技大学）；Department of Computer Science and Engineering, Southern University of Science and Technology（计算机科学与工程系，南方科技大学）；VIVO
Disentangling content and style information of an image has played an important role in recent success in image translation. In this setting, how to inject given style into an input image containing its own content is an important issue, but existing methods followed relatively simple approaches, leaving room for improvement especially when incorporating significant style changes. In response, we propose an advanced normalization technique based on adaptive convolution (AdaCoN), in order to properly impose style information into the content of an input image. In detail, after locally standardi…
DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion
University of Electronic Science and Technology of China（电子科学与技术大学）；Shenzhen Institute for Advanced Study, UESTC（深圳先进研究院）；Computational Mathematics, Science, & Engineering at Michigan State University (MSU)（密歇根州立大学计算数学、科学与工程）
Convolutional neural networks (CNNs) have been not only widespread but also achieved noticeable results on numerous applications including image classification, restoration, and generation. Although the weight-sharing property of convolutions makes them widely adopted in various tasks, its content-agnostic characteristic can also be considered a major drawback. To solve this problem, in this paper, we propose a novel operation, called pixel adaptive kernel attention (PAKA). PAKA provides directivity to the filter weights by multiplying spatially varying attention from learnable features. The p…
Task-Adaptive 3D Cross-Field MRI Translation via Field-Conditioned Content-Style Pretraining
School of Integrated Circuits and Electronics, Beijing Institute of Technology（北京理工大学集成电路与电子学院）；School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University（清华大学医学院生物医学工程学院）；Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）
This paper addresses the problem of manipulating images using natural language description. Our task aims to semantically modify visual attributes of an object in an image according to the text describing the new visual appearance. Although existing methods synthesize images having new attributes, they do not fully preserve text-irrelevant contents of the original image. In this paper, we propose the text-adaptive generative adversarial network (TAGAN) to generate semantically manipulated images while preserving text-irrelevant contents. The key to our method is the text-adaptive discriminator…
ROI-Gated SAHI: Content-Adaptive Slicing-Based Inference for Efficient Object Detection
Faculty of Information Technology, City University, Malaysia（马来西亚城市大学信息技术学院）；Kyung Hee University（庆熙大学）；National University of Sciences and Technology, Pakistan（巴基斯坦国家科技大学）；School of Computing and Artificial Intelligence, Sunway University, Malaysia（马来西亚双威大学计算与人工智能学院）；King Fahd University of Petroleum and Minerals, Saudi Arabia（沙特阿拉伯法赫德国王石油和矿产大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence, Saudi Arabia（沙特阿拉伯SDAIA-KFUPM人工智能联合研究中心）
Sun Yat-sen University（中山大学）；Hisilicon Technologies Co., Ltd.（海思技术有限公司）；Queen Mary University of London（伦敦玛丽女王大学）
Sketch portrait generation benefits a wide range of applications such as digital entertainment and law enforcement. Although plenty of efforts have been dedicated to this task, several issues still remain unsolved for generating vivid and detail-preserving personal sketch portraits. For example, quite a few artifacts may exist in synthesizing hairpins and glasses, and textural details may be lost in the regions of hair or mustache. Moreover, the generalization ability of current systems is somewhat limited since they usually require elaborately collecting a dictionary of examples or carefully …
Over the past few years, image-to-image (I2I) translation methods have been proposed to translate a given image into diverse outputs. Despite the impressive results, they mainly focus on the I2I translation between two domains, so the multi-domain I2I translation still remains a challenge. To address this problem, we propose a novel multi-domain unsupervised image-to-image translation (MDUIT) framework that leverages the decomposed content feature and appearance adaptive convolution to translate an image into a target appearance while preserving the given geometric content. We also exploit a c…
