Recently, the performance of single image super-resolution (SR) has been significantly improved with powerful networks. However, these networks are developed for image SR with a single specific integer scale (e.g., x2;x3,x4), and cannot be used for non-integer and asymmetric SR. In this paper, we propose to learn a scale-arbitrary image SR network from scale-specific networks. Specifically, we propose a plug-in module for existing SR networks to perform scale-arbitrary SR, which consists of multiple scale-aware feature adaption blocks and a scale-aware upsampling layer. Moreover, we introduce a scale-aware knowledge transfer paradigm to transfer knowledge from scale-specific networks to the scale-arbitrary network. Our plug-in module can be easily adapted to existing networks to achieve scale-arbitrary SR. These networks plugged with our module can achieve promising results for non-integer and asymmetric SR while maintaining state-of-the-art performance for SR with integer scale factors. Besides, the additional computational and memory cost of our module is very small.
University of Science and Technology of China（中国科学技术大学）；Center for Research on Intelligent Perception and Computing, NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室智能感知与计算研究中心）；Tsinghua University（清华大学）；Megvii Inc (Face++)（旷视科技（Face++））
Recent research on super-resolution has achieved great success due to the development of deep convolutional neural networks (DCNNs). However, super-resolution of arbitrary scale factor has been ignored for a long time. Most previous researchers regard super-resolution of different scale factors as independent tasks. They train a specific model for each scale factor which is inefficient in computing, and prior work only take the super-resolution of several integer scale factors into consideration. In this work, we propose a novel method called Meta-SR to firstly solve super-resolution of arbitr…
Tianjin University（天津大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Information Science and Technology（南京信息工程大学）
Implicit neural representation (INR) is a popular approach for arbitrary-scale image super-resolution (SR), as a key component of INR, position encoding improves its representation ability. Motivated by position encoding, we propose orthogonal position encoding (OPE) - an extension of position encoding - and an OPE-Upscale module to replace the INR-based upsampling module for arbitrary-scale image super-resolution. Same as INR, our OPE-Upscale Module takes 2D coordinates and latent code as inputs; however it does not require training parameters. This parameter-free feature allows the OPE-Upsca…
Deep CNNs have achieved significant successes in image processing and its applications, including single image super-resolution (SR). However, conventional methods still resort to some predetermined integer scaling factors, e.g., x2 or x4. Thus, they are difficult to be applied when arbitrary target resolutions are required. Recent approaches extend the scope to real-valued upsampling factors, even with varying aspect ratios to handle the limitation. In this paper, we propose the SRWarp framework to further generalize the SR tasks toward an arbitrary image transformation. We interpret the trad…
This paper addresses 2 challenging tasks: improving the quality of low resolution facial images and accurately locating the facial landmarks on such poor resolution images. To this end, we make the following 5 contributions: (a) we propose Super-FAN: the very first end-to-end system that addresses both tasks simultaneously, i.e. both improves face resolution and detects the facial landmarks. The novelty or Super-FAN lies in incorporating structural information in a GAN-based super-resolution algorithm via integrating a sub-network for face alignment through heatmap regression and optimizing a …
The 360°imaging has recently gained great attention; however, its angular resolution is relatively lower than that of a narrow field-of-view (FOV) perspective image as it is captured by using fisheye lenses with the same sensor size. Therefore, it is beneficial to super-resolve a 360°image. Some attempts have been made but mostly considered the equirectangular projection (ERP) as one of the way for 360°image representation despite of latitude-dependent distortions. In that case, as the output high-resolution(HR) image is always in the same ERP format as the low-resolution (LR) input, another i…
Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）；Peng Cheng Laboratory（鹏城实验室）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
While deep neural networks (DNN) based single image super-resolution (SISR) methods are rapidly gaining popularity, they are mainly designed for the widely-used bicubic degradation, and there still remains the fundamental challenge for them to super-resolve low-resolution (LR) image with arbitrary blur kernels. In the meanwhile, plug-and-play image restoration has been recognized with high flexibility due to its modular structure for easy plug-in of denoiser priors. In this paper, we propose a principled formulation and framework by extending bicubic degradation based deep SISR with the help o…
Scale-Invariant Adversarial Attack against Arbitrary-scale Super-resolution
Nanyang Technological University（南洋理工大学）；East China Normal University（华东师范大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；New York University（纽约大学）
The recent success of NeRF and other related implicit neural representation methods has opened a new path for continuous image representation, where pixel values no longer need to be looked up from stored discrete 2D arrays but can be inferred from neural network models on a continuous spatial domain. Although the recent work LIIF has demonstrated that such novel approaches can achieve good performance on the arbitrary-scale super-resolution task, their upscaled images frequently show structural distortion due to the inaccurate prediction of high-frequency textures. In this work, we propose Ul…
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Existing models on super-resolution often specialized for one scale, fundamentally limiting their use in practical scenarios. In this paper, we aim to develop a general plugin that can be inserted into existing super-resolution models, conveniently augmenting their ability towards Arbitrary Resolution Image Scaling, thus termed ARIS. We make the following contributions: (i) we propose a transformer-based plugin module, which uses spatial coordinates as query, iteratively attend the low-resolution image feature through cross-attention, and output visual feature for the queried spatial location,…
Task-Aware Dynamic Transformer for Efficient Arbitrary-Scale Image Super-Resolution
School of Statistics and Data Science, Nankai University（南开大学统计与数据科学学院）；College of Computer Science, Nankai University（南开大学计算机学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；Tencent Data Platform（腾讯云与智慧产业事业群数据平台部）；Zhejiang University（浙江大学）；Systems Engineering Research Institute, China State Shipbuilding Corporation Limited（中国船舶重工集团公司系统工程研究院）；Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）广东省大数据计算重点实验室）
Arbitrary-Scale Video Super-Resolution with Structural and Textural Priors
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Jiangxi University of Finance and Economics（江西财经大学）
National Tsing Hua University（国立清华大学）；MediaTek Inc.（联发科技股份有限公司）
Implicit neural representation has recently shown a promising ability in representing images with arbitrary resolutions. In this paper, we present a Local Implicit Transformer (LIT), which integrates the attention mechanism and frequency encoding technique into a local implicit image function. We design a cross-scale local attention block to effectively aggregate local features. To further improve representative power, we propose a Cascaded LIT (CLIT) that exploits multi-scale features, along with a cumulative training strategy that gradually increases the upsampling scales during training. We…
OmniScaleSR: Unleashing Scale-Controlled Diffusion Prior for Faithful and Realistic Arbitrary-Scale Image Super-Resolution
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学信息科学与电子工程学院）；College of Information, Mechanical and Electrical Engineering, Shanghai Normal University（上海师范大学信息、机械与电气工程学院）；Transsion in China（中国Transsion）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能教育部重点实验室）
School of Mathematics and Statistics, Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University（数学与统计学学院， 教育部智能网络与网络安全重点实验室， 西安交通大学）；Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University（教育部智能网络与网络安全重点实验室， 西安交通大学）；Macao Institute of Systems Engineering, Macau University of Science and Technology（澳门系统工程研究所， 澳门科技大学）
Conventional social media platforms usually downscale the HR images to restrict their resolution to a specific size for saving transmission/storage cost, which leads to the super-resolution (SR) being highly ill-posed. Recent invertible image downscaling methods jointly model the downscaling/upscaling problems and achieve significant improvements. However, they only consider fixed integer scale factors that cannot downscale HR images with various resolutions to meet the resolution restriction of social media platforms. In this paper, we propose a scale-Arbitrary Invertible image Downscaling Ne…
Self-supervised arbitrary scale super-resolution framework for anisotropic MRI
ShanghaiTech University（上海科技大学）；Southwest Hospital, Army Medical University (Third Military Medical University)（陆军军医大学（第三军医大学）西南医院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；Central Research Institute, UIH Group（联影集团中央研究院）；Shanghai Jiao Tong University（上海交通大学）
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University at Buffalo（布法罗大学）；Syracuse University（雪城大学）
Hyperspectral image (HSI) with narrow spectral bands can capture rich spectral information, but it sacrifices its spatial resolution in the process. Many machine-learning-based HSI super-resolution (SR) algorithms have been proposed recently. However, one of the fundamental limitations of these approaches is that they are highly dependent on image and camera settings and can only learn to map an input HSI with one specific setting to an output HSI with another. However, different cameras capture images with different spectral response functions and bands numbers due to the diversity of HSI cam…
BasicAVSR: Arbitrary-Scale Video Super-Resolution via Image Priors and Enhanced Motion Compensation
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Tianjin Key Laboratory of Machine Learning, College of Intelligence and Computing, Tianjin University（天津机器学习重点实验室，天津大学智能计算学院）
University of Technology Sydney（悉尼科技大学）；Bytedance（字节跳动）
Deep convolutional neural networks have significantly improved the peak signal-to-noise ratio of SuperResolution (SR). However, image viewer applications commonly allow users to zoom the images to arbitrary magnification scales, thus far imposing a large number of required training scales at a tremendous computational cost. To obtain a more computationally efficient model for arbitrary scale SR, this paper employs a Laplacian pyramid method to reconstruct any-scale high-resolution (HR) images using the high-frequency image details in a Laplacian Frequency Representation. For SR of small-scales…
Image super-resolution (SR) is a technique to recover lost high-frequency information in low-resolution (LR) images. Spatial-domain information has been widely exploited to implement image SR, so a new trend is to involve frequency-domain information in SR tasks. Besides, image SR is typically application-oriented and various computer vision tasks call for image arbitrary magnification. Therefore, in this paper, we study image features in the frequency domain to design a novel scale-arbitrary image SR network. First, we statistically analyze LR-HR image pairs of several datasets under differen…
Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The Hong Kong Polytechnic University（香港理工大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
Recent attempts at Super-Resolution for medical images used deep learning techniques such as Generative Adversarial Networks (GANs) to achieve perceptually realistic single image Super-Resolution. Yet, they are constrained by their inability to generalise to different scale factors. This involves high storage and energy costs as every integer scale factor involves a separate neural network. A recent paper has proposed a novel meta-learning technique that uses a Weight Prediction Network to enable Super-Resolution on arbitrary scale factors using only a single neural network. In this paper, we …
Recently, image super-resolution has been widely studied and achieved significant progress by leveraging the power of deep convolutional neural networks. However, there has been limited advancement in video super-resolution (VSR) due to the complex temporal patterns in videos. In this paper, we investigate how to adapt state-of-the-art methods of image super-resolution for video super-resolution. The proposed adapting method is straightforward. The information among successive frames is well exploited, while the overhead on the original image super-resolution method is negligible. Furthermore,…
An efficient dual-branch framework via implicit self-texture enhancement for arbitrary-scale histopathology image super-resolution
Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）
ShanghaiTech University（上海科技大学）；Ren Ji Hospital, School of Medicine, Shanghai Jiao Tong University（上海交通大学医学院仁济医院）；Shanghai Jiao Tong University（上海交通大学）
High Resolution (HR) medical images provide rich anatomical structure details to facilitate early and accurate diagnosis. In MRI, restricted by hardware capacity, scan time, and patient cooperation ability, isotropic 3D HR image acquisition typically requests long scan time and, results in small spatial coverage and low SNR. Recent studies showed that, with deep convolutional neural networks, isotropic HR MR images could be recovered from low-resolution (LR) input via single image super-resolution (SISR) algorithms. However, most existing SISR methods tend to approach a scale-specific projecti…
Magnetic resonance (MR) images collected in 2D clinical protocols typically have large inter-slice spacing, resulting in high in-plane resolution and reduced through-plane resolution. Super-resolution technique can enhance the through-plane resolution of MR images to facilitate downstream visualization and computer-aided diagnosis. However, most existing works train the super-resolution network at a fixed scaling factor, which is not friendly to clinical scenes of varying inter-slice spacing in MR scanning. Inspired by the recent progress in implicit neural representation, we propose a Spatial…
School of Computer and Software Engineering, Xihua University（西华大学计算机与软件工程学院）
Image super-resolution technology is the process of obtaining high-resolution images from one or more low-resolution images. With the development of deep learning, image super-resolution technology based on deep learning method is emerging. This paper reviews the research progress of the application of depth learning method in the field of image super-resolution, introduces this kind of super-resolution work from several aspects, and looks forward to the further application of depth learning method in the field of image super-resolution. By collecting and counting the relevant literature on th…
M^3:Manipulation Mask Manufacturer for Arbitrary-Scale Super-Resolution Mask
College of Computer Science, Sichuan University（四川大学计算机学院）；University of Macao（澳门大学）；Engineering Research Center of Machine Learning and Industry Intelligence, MOE, China（教育部机器学习与产业智能工程研究中心）
Depth image super-resolution is an extremely challenging task due to the information loss in sub-sampling. Deep convolutional neural network have been widely applied to color image super-resolution. Quite surprisingly, this success has not been matched to depth super-resolution. This is mainly due to the inherent difference between color and depth images. In this paper, we bridge up the gap and extend the success of deep convolutional neural network to depth super-resolution. The proposed deep depth super-resolution method learns the mapping from a low-resolution depth image to a high resoluti…
Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations
Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong（香港中文大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong Baptist University（香港浸会大学）
Joint Degradation-Aware Arbitrary-Scale Super-Resolution for Variable-Rate Extreme Image Compression
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学信息科学与电子工程学院）；Department of Electronic Engineer and Information Science, University of Science and Technology of China（中国科学技术大学电子工程与信息科学系）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能教育部重点实验室）
