In this paper, we present a Hybrid Spectral Denoising Transformer (HSDT) for hyperspectral image denoising. Challenges in adapting transformer for HSI arise from the capabilities to tackle existing limitations of CNN-based methods in capturing the global and local spatial-spectral correlations while maintaining efficiency and flexibility. To address these issues, we introduce a hybrid approach that combines the advantages of both models with a Spatial-Spectral Separable Convolution (S3Conv), Guided Spectral Self-Attention (GSSA), and Self-Modulated Feed-Forward Network (SM-FFN). Our S3Conv works as a lightweight alternative to 3D convolution, which extracts more spatial-spectral correlated features while keeping the flexibility to tackle HSIs with an arbitrary number of bands. These features are then adaptively processed by GSSA which per-forms 3D self-attention across the spectral bands, guided by a set of learnable queries that encode the spectral signatures. This not only enriches our model with powerful capabilities for identifying global spectral correlations but also maintains linear complexity. Moreover, our SM-FFN proposes the self-modulation that intensifies the activations of more informative regions, which further strengthens the aggregated features. Extensive experiments are conducted on various datasets under both simulated and real-world noise, and it shows that our HSDT significantly outperforms the existing state-of-the-art methods while maintaining low computational overhead. Code is at https: //github.com/Zeqiang-Lai/HSDT.
Birla Institute of Technology Science Pilani（比尔拉理工学院皮拉尼分校）
In this work, we present Eformer - Edge enhancement based transformer, a novel architecture that builds an encoder-decoder network using transformer blocks for medical image denoising. Non-overlapping window-based self-attention is used in the transformer block that reduces computational requirements. This work further incorporates learnable Sobel-Feldman operators to enhance edges in the image and propose an effective way to concatenate them in the intermediate layers of our architecture. The experimental analysis is conducted by comparing deterministic learning and residual learning for the …
Heterogeneous window transformer for image denoising
Northwestern Polytechnical University（西北工业大学）；National Tsing Hua University（国立清华大学）；Xidian University（西安电子科技大学）；Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
Hyperspectral image (HSI) denoising is a crucial preprocessing procedure for the subsequent HSI applications. Unfortunately, though witnessing the development of deep learning in HSI denoising area, existing convolution-based methods face the trade-off between computational efficiency and capability to model non-local characteristics of HSI. In this paper, we propose a Spatial-Spectral Transformer (SST) to alleviate this problem. To fully explore intrinsic similarity characteristics in both spatial dimension and spectral dimension, we conduct non-local spatial self-attention and global spectra…
Nanjing University of Science and Technology（南京理工大学）；University of Macau（澳门大学）；Griffith University（格里菲斯大学）；Zhejiang University（浙江大学）
Deep learning (DL) based hyperspectral images (HSIs) denoising approaches directly learn the nonlinear mapping between observed noisy images and underlying clean images. They normally do not consider the physical characteristics of HSIs, therefore making them lack of interpretability that is key to understand their denoising mechanism.. In order to tackle this problem, we introduce a novel model guided interpretable network for HSI denoising. Specifically, fully considering the spatial redundancy, spectral low-rankness and spectral-spatial properties of HSIs, we first establish a subspace base…
Fully supervised deep-learning based denoisers are currently the most performing image denoising solutions. However, they require clean reference images. When the target noise is complex, e.g. composed of an unknown mixture of primary noises with unknown intensity, fully supervised solutions are limited by the difficulty to build a suited training set for the problem. This paper proposes a gradual denoising strategy that iteratively detects the dominating noise in an image, and removes it using a tailored denoiser. The method is shown to keep up with state of the art blind denoisers on mixture…
Xidian University（西安电子科技大学）；West Virginia University（西弗吉尼亚大学）
Neural architecture search (NAS) has recently reshaped our understanding on various vision tasks. Similar to the success of NAS in high-level vision tasks, it is possible to find a memory and computationally efficient solution via NAS with highly competent denoising performance. However, the optimization gap between the super-network and the sub-architectures has remained an open issue in both low-level and high-level vision. In this paper, we present a novel approach to filling in this gap by connecting model-guided design with NAS (MoD-NAS) and demonstrate its application into image denoisin…
Low dose computed tomography is a mainstream for clinical applications. How-ever, compared to normal dose CT, in the low dose CT (LDCT) images, there are stronger noise and more artifacts which are obstacles for practical applications. In the last few years, convolution-based end-to-end deep learning methods have been widely used for LDCT image denoising. Recently, transformer has shown superior performance over convolution with more feature interactions. Yet its ap-plications in LDCT denoising have not been fully cultivated. Here, we propose a convolution-free T2T vision transformer-based Enc…
Pan-denoising: Guided Hyperspectral Image Denoising via Weighted Represent Coefficient Total Variation
School of Mathematics and Statistics, Northwestern Polytechnical University（西北工业大学数学与统计学院）；Research and Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
National Chung Hsing University（国立中兴大学）；National Taichung University of Science and Technology（国立台中科技大学）
Image restoration is a challenging ill-posed problem which also has been a long-standing issue. In the past few years, the convolution neural networks (CNNs) almost dominated the computer vision and had achieved considerable success in different levels of vision tasks including image restoration. However, recently the Swin Transformer-based model also shows impressive performance, even surpasses the CNN-based methods to become the state-of-the-art on high-level vision tasks. In this paper, we proposed a restoration model called SUNet which uses the Swin Transformer layer as our basic block and…
School of Statistics and Data Science, Nankai University（南开大学统计与数据科学学院）；The First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院）；Shanxi Engineering Research Center of Computational Imaging and Medical Intelligence（山西省计算成像与医学智能工程研究中心）
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）；Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Central South University（中南大学）；National Tsing Hua University（国立清华大学）
EWT: Efficient Wavelet-Transformer for Single Image Denoising
Shanghai University（上海大学）；Nanjing University of Science and Technology（南京理工大学）；Xidian University（西安电子科技大学）；Beijing Electronic Science & Technology Institute（北京电子科技学院）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Soochow University（苏州大学）；The Chinese University of Hong Kong（香港中文大学）
The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
Most of existing image denoising methods learn image priors from either external data or the noisy image itself to remove noise. However, priors learned from external data may not be adaptive to the image to be denoised, while priors learned from the given noisy image may not be accurate due to the interference of corrupted noise. Meanwhile, the noise in real-world noisy images is very complex, which is hard to be described by simple distributions such as Gaussian distribution, making real-world noisy image denoising a very challenging problem. We propose to exploit the information in both ext…
University of Science and Technology Beijing（北京科技大学）；Beijing Jiaotong University（北京交通大学）
Image denoising is an important low-level computer vision task, which aims to reconstruct a noise-free and high-quality image from a noisy image. With the development of deep learning, convolutional neural network (CNN) has been gradually applied and achieved great success in image denoising, image compression, image enhancement, etc. Recently, Transformer has been a hot technique, which is widely used to tackle computer vision tasks. However, few Transformer-based methods have been proposed for low-level vision tasks. In this paper, we proposed an image denoising network structure based on Tr…
Spach Transformer: Spatial and Channel-wise Transformer Based on Local and Global Self-attentions for PET Image Denoising
Gordon Center for Medical Imaging（戈登医学影像中心）；Center for Advanced Medical Computing and Analysis（高级医学计算与分析中心）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Florida（佛罗里达大学）；University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Johns Hopkins University（约翰斯·霍普金斯大学）
针对 PET 图像噪声及三维全局自注意力计算成本高的问题，提出融合局部与全局 MSA 的空间通道编解码 Transformer，并在多示踪剂数据集上验证其去噪性能优于现有深度学习架构。
SGDFormer: One-stage Transformer-based Architecture for Cross-Spectral Stereo Image Guided Denoising
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与工程学院）；Ningbo Research Institute, Zhejiang University（浙江大学宁波研究院）；Zhejiang GongShang University（浙江工商大学）
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）；Sun Yat-sen University（中山大学）；Nanjing University of Science and Technology（南京理工大学）
Real-world image denoising is a practical image restoration problem that aims to obtain clean images from in-the-wild noisy inputs. Recently, the Vision Transformer (ViT) has exhibited a strong ability to capture long-range dependencies, and many researchers have attempted to apply the ViT to image denoising tasks. However, a real-world image is an isolated frame that makes the ViT build long-range dependencies based on the internal patches, which divides images into patches, disarranges noise patterns and damages gradient continuity. In this article, we propose to resolve this issue by using …
Xformer: Hybrid X-Shaped Transformer for Image Denoising
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Sydney（悉尼大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；ETH Zürich（苏黎世联邦理工学院）
The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）
We propose an effective Structural Prior guided Generative Adversarial Transformer (SPGAT) to solve low-light image enhancement. Our SPGAT mainly contains a generator with two discriminators and a structural prior estimator (SPE). The generator is based on a U-shaped Transformer which is used to explore non-local information for better clear image restoration. The SPE is used to explore useful structures from images to guide the generator for better structural detail estimation. To generate more realistic images, we develop a new structural prior guided adversarial learning method by building …
Hamamatsu Photonics K. K.（滨松光子学株式会社）；Hamamatsu University School of Medicine（滨松医科大学）
Although supervised convolutional neural networks (CNNs) often outperform conventional alternatives for denoising positron emission tomography (PET) images, they require many low- and high-quality reference PET image pairs. Herein, we propose an unsupervised 3D PET image denoising method based on an anatomical information-guided attention mechanism. The proposed magnetic resonance-guided deep decoder (MR-GDD) utilizes the spatial details and semantic features of MR-guidance image more effectively by introducing encoder-decoder and deep decoder subnetworks. Moreover, the specific shapes and pat…
In this paper, we propose a novel and efficient CNN-based framework that leverages local and global context information for image denoising. Due to the limitations of convolution itself, the CNN-based method is generally unable to construct an effective and structured global feature representation, usually called the long-distance dependencies in the Transformer-based method. To tackle this problem, we introduce the matrix decomposition module(MD) in the network to establish the global context feature, comparable to the Transformer based method performance. Inspired by the design of multi-stag…
VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising
The Chinese University of Hong Kong（香港中文大学）；National University of Defense Technology（国防科技大学）；City University of Hong Kong（香港城市大学）；Nanjing University of Science and Technology（南京理工大学）
A TRPCA-Inspired Deep Unfolding Network for Hyperspectral Image Denoising via Thresholded t-SVD and Top-K Sparse Transformer
College of Computer Science and Engineering, Shandong University of Science and Technology（计算机科学与工程学院，山东科技大学）；Gosci Information Technology Co.,Ltd.（Gosci信息技术有限公司）
School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学学院）；Department of Computer and Information Science, Faculty of Science and Technology, University of Macau（澳门大学计算机与信息科学系）；Hubei Center for Applied Mathematics, Wuhan University（湖北应用数学中心，武汉大学）；Hubei Key Laboratory of Computational Science, Wuhan University（湖北计算科学重点实验室，武汉大学）
Inception Institute of AI（人工智能起源研究所）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Monash University（莫纳什大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to ap…
SwinIA: Self-Supervised Blind-Spot Image Denoising without Convolutions
Institute of Computer Science, University of Tartu（塔尔图大学计算机科学学院）；CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用技术大学CAIRO机构）
University of Massachusetts, Lowell（马萨诸塞大学洛厄尔分校）；Weill Cornell Medicine, Cornell University（康奈尔大学威尔康奈尔医学院）
Low-dose computed tomography (LDCT) denoising is an important problem in CT research. Compared to the normal dose CT (NDCT), LDCT images are subjected to severe noise and artifacts. Recently in many studies, vision transformers have shown superior feature representation ability over convolutional neural networks (CNNs). However, unlike CNNs, the potential of vision transformers in LDCT denoising was little explored so far. To fill this gap, we propose a Convolution-free Token2Token Dilated Vision Transformer for low-dose CT denoising. The CTformer uses a more powerful token rearrangement to en…
Low-Dose Computed Tomography (LDCT) technique, which reduces the radiation harm to human bodies, is now attracting increasing interest in the medical imaging field. As the image quality is degraded by low dose radiation, LDCT exams require specialized reconstruction methods or denoising algorithms. However, most of the recent effective methods overlook the inner-structure of the original projection data (sinogram) which limits their denoising ability. The inner-structure of the sinogram represents special characteristics of the data in the sinogram domain. By maintaining this structure while d…
McGill University（麦吉尔大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
In recent years, self-supervised denoising methods have shown impressive performance, which circumvent painstaking collection procedure of noisy-clean image pairs in supervised denoising methods and boost denoising applicability in real world. One of well-known self-supervised denoising strategies is the blind-spot training scheme. However, a few works attempt to improve blind-spot based self-denoiser in the aspect of network architecture. In this paper, we take an intuitive view of blind-spot strategy and consider its process of using neighbor pixels to predict manipulated pixels as an inpain…
ETH Zürich（苏黎世联邦理工学院）；University of Wurzburg（维尔茨堡大学）；KU Leuven（鲁汶大学）
Video denoising aims at removing noise from videos to recover clean ones. Some existing works show that optical flow can help the denoising by exploiting the additional spatial-temporal clues from nearby frames. However, the flow estimation itself is also sensitive to noise, and can be unusable under large noise levels. To this end, we propose a new multi-scale refined optical flow-guided video denoising method, which is more robust to different noise levels. Our method mainly consists of a denoising-oriented flow refinement (DFR) module and a flow-guided mutual denoising propagation (FMDP) mo…
MDAA-Diff: CT-Guided Multi-Dose Adaptive Attention Diffusion Model for PET Denoising
School of Biomedical Engineering, Southern Medical University（南方医科大学生物医学工程学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Nanfang PET Center, Nanfang Hospital Southern Medical University（南方医科大学南芳医院PET中心）
Low-dose computed tomography (LDCT) reduces the X-ray radiation but compromises image quality with more noises and artifacts. A plethora of transformer models have been developed recently to improve LDCT image quality. However, the success of a transformer model relies on a large amount of paired noisy and clean data, which is often unavailable in clinical applications. In computer vision and natural language processing fields, masked autoencoders (MAE) have been proposed as an effective label-free self-pretraining method for transformers, due to its excellent feature representation ability. H…
Cycle-guided Denoising Diffusion Probability Model for 3D Cross-modality MRI Synthesis
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
Video denoising aims to recover high-quality frames from the noisy video. While most existing approaches adopt convolutional neural networks~(CNNs) to separate the noise from the original visual content, however, CNNs focus on local information and ignore the interactions between long-range regions in the frame. Furthermore, most related works directly take the output after basic spatio-temporal denoising as the final result, leading to neglect the fine-grained denoising process. In this paper, we propose a Dual-stage Spatial-Channel Transformer for coarse-to-fine video denoising, which inheri…
