Self-supervised video denoising has seen decent progress through the use of blind spot networks. However, under their blind spot constraints, previous self-supervised video denoising methods suffer from significant information loss and texture destruction in either the whole reference frame or neighbor frames, due to their inadequate consideration of the receptive field. Moreover, the limited number of available neighbor frames in previous methods leads to the discarding of distant temporal information. Nonetheless, simply adopting existing recurrent frameworks does not work, since they easily break the constraints on the receptive field imposed by self-supervision. In this paper, we propose RDRF for self-supervised video denoising, which not only fully exploits both the reference and neighbor frames with a denser receptive field, but also better leverages the temporal information from both local and distant neighbor features. First, towards a comprehensive utilization of information from both reference and neighbor frames, RDRF realizes a denser receptive field by taking more neighbor pixels along the spatial and temporal dimensions. Second, it features a self-supervised recurrent video denoising framework, which concurrently integrates distant and near-neighbor temporal features. This enables long-term bidirectional information aggregation, while mitigating error accumulation in the plain recurrent framework. Our method exhibits superior performance on both synthetic and real video denoising datasets. Codes will be available at https://github.com/Wang-XIaoDingdd/RDRF.
Variations of deep neural networks such as convolutional neural network (CNN) have been successfully applied to image denoising. The goal is to automatically learn a mapping from a noisy image to a clean image given training data consisting of pairs of noisy and clean images. Most existing CNN models for image denoising have many layers. In such cases, the models involve a large amount of parameters and are computationally expensive to train. In this paper, we develop a dilated residual CNN for Gaussian image denoising. Compared with the recently proposed residual denoiser, our method can achi…
Classical image denoising methods utilize the non-local self-similarity principle to effectively recover image content from noisy images. Current state-of-the-art methods use deep convolutional neural networks (CNNs) to effectively learn the mapping from noisy to clean images. Deep denoising CNNs manifest a high learning capacity and integrate non-local information owing to the large receptive field yielded by numerous cascade of hidden layers. However, deep networks are also computationally complex and require large data for training. To address these issues, this study draws the focus on the…
The non-local self-similarity property of natural images has been exploited extensively for solving various image processing problems. When it comes to video sequences, harnessing this force is even more beneficial due to the temporal redundancy. In the context of image and video denoising, many classically-oriented algorithms employ self-similarity, splitting the data into overlapping patches, gathering groups of similar ones and processing these together somehow. With the emergence of convolutional neural networks (CNN), the patch-based framework has been abandoned. Most CNN denoisers operat…
Harbin Institute of Technology（哈尔滨工业大学）；Sun Yat-Sen University（中山大学）
The tradeoff between receptive field size and efficiency is a crucial issue in low level vision. Plain convolutional networks (CNNs) generally enlarge the receptive field at the expense of computational cost. Recently, dilated filtering has been adopted to address this issue. But it suffers from gridding effect, and the resulting receptive field is only a sparse sampling of input image with checkerboard patterns. In this paper, we present a novel multi-level wavelet CNN (MWCNN) model for better tradeoff between receptive field size and computational efficiency. With the modified U-Net architec…
Video streams are delivered continuously to save the cost of storage and device memory. Real-time denoising algorithms are typically adopted on the user device to remove the noise involved during the shooting and transmission of video streams. However, sliding-window-based methods feed multiple input frames for a single output and lack computation efficiency. Recent multi-output inference works propagate the bidirectional temporal feature with a parallel or recurrent framework, which either suffers from performance drops on the temporal edges of clips or can not achieve online inference. In th…
Imperial College London（帝国理工学院）；Fudan University（复旦大学）
Dilated convolutions are widely used in deep semantic segmentation models as they can enlarge the filters' receptive field without adding additional weights nor sacrificing spatial resolution. However, as dilated convolutional filters do not possess positional knowledge about the pixels on semantically meaningful contours, they could lead to ambiguous predictions on object boundaries. In addition, although dilating the filter can expand its receptive field, the total number of sampled pixels remains unchanged, which usually comprises a small fraction of the receptive field's total area. Inspir…
Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；Computer Vision Center, Barcelona（巴塞罗那计算机视觉中心）；Facebook AI Research（Facebook人工智能研究）
State-of-the-art semantic segmentation approaches increase the receptive field of their models by using either a downsampling path composed of poolings/strided convolutions or successive dilated convolutions. However, it is not clear which operation leads to best results. In this paper, we systematically study the differences introduced by distinct receptive field enlargement methods and their impact on the performance of a novel architecture, called Fully Convolutional DenseResNet (FC-DRN). FC-DRN has a densely connected backbone composed of residual networks. Following standard image segment…
We describe a novel method for training high-quality image denoising models based on unorganized collections of corrupted images. The training does not need access to clean reference images, or explicit pairs of corrupted images, and can thus be applied in situations where such data is unacceptably expensive or impossible to acquire. We build on a recent technique that removes the need for reference data by employing networks with a "blind spot" in the receptive field, and significantly improve two key aspects: image quality and training efficiency. Our result quality is on par with state-of-t…
Harbin Institute of Technology（哈尔滨工业大学）；Changzhi University（长治学院）
In computer vision, convolutional networks (CNNs) often adopts pooling to enlarge receptive field which has the advantage of low computational complexity. However, pooling can cause information loss and thus is detrimental to further operations such as features extraction and analysis. Recently, dilated filter has been proposed to trade off between receptive field size and efficiency. But the accompanying gridding effect can cause a sparse sampling of input images with checkerboard patterns. To address this problem, in this paper, we propose a novel multi-level wavelet CNN (MWCNN) model to ach…
Adaptive Dynamic Filtering Network for Image Denoising
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)（合肥工业大学计算机与信息工程学院）；Intelligent Interconnected Systems Laboratory of Anhui Province (HFUT)（安徽省智能互联系统实验室（合肥工业大学））；Guangxi Academy of Sciences（广西科学院）；Intelligent Manufacturing Institute of HFUT（合肥工业大学智能制造研究院）
MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）；School of Computer Science and Technology, Shenzhen Campus of Harbin Institute of Technology（哈尔滨工业大学深圳校区计算机科学与技术学院）；School of Engineering and Computer Science, Australian National University（澳大利亚国立大学工程与计算机科学学院）；Hong Kong University of Science and Technology（香港科技大学）
CIM-NET: A Video Denoising Deep Neural Network Model Optimized for Computing-in-Memory Architectures
China Mobile Communications Group Co.,Ltd.（中国移动通信集团有限公司）；ZGC Institute of Ubiquitous-X Innovation and Applications（Ubiquitous-X创新与应用研究院）；China Mobile Research Institute（中国移动研究院）
Huawei Technologies Ltd（华为技术有限公司）；Queen Mary University of London（伦敦玛丽女王大学）；University of Oxford（牛津大学）
Recurrent models are a popular choice for video enhancement tasks such as video denoising or super-resolution. In this work, we focus on their stability as dynamical systems and show that they tend to fail catastrophically at inference time on long video sequences. To address this issue, we (1) introduce a diagnostic tool which produces input sequences optimized to trigger instabilities and that can be interpreted as visualizations of temporal receptive fields, and (2) propose two approaches to enforce the stability of a model during training: constraining the spectral norm or constraining the…
Spach Transformer: Spatial and Channel-wise Transformer Based on Local and Global Self-attentions for PET Image Denoising
Gordon Center for Medical Imaging（戈登医学影像中心）；Center for Advanced Medical Computing and Analysis（高级医学计算与分析中心）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Florida（佛罗里达大学）；University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Johns Hopkins University（约翰斯·霍普金斯大学）
针对 PET 图像噪声及三维全局自注意力计算成本高的问题，提出融合局部与全局 MSA 的空间通道编解码 Transformer，并在多示踪剂数据集上验证其去噪性能优于现有深度学习架构。
Adapting MIMO video restoration networks to low latency constraints
Université Paris-Saclay（巴黎萨克雷大学）；ENS Paris-Saclay（巴黎萨克雷高等师范学院）；CNRS（法国国家科学研究中心）；Centre Borelli（博雷利中心）；Facultad de Ingeniería, Universidad de la República（乌拉圭共和国大学工程学院）；Thales LAS（泰雷兹实验室）；City University of Hong Kong（香港城市大学）；Université d’Orléans（奥尔良大学）；Institut Denis Poisson（德尼·泊松研究所）；Pompeu Fabra University（庞培法布拉大学）
