Image restoration aims to recover high-quality images from their corrupted counterparts. Many existing methods primarily focus on the spatial domain, neglecting the understanding of frequency variations and ignoring the impact of implicit noise in skip connections. In this paper, we introduce a multi-scale frequency selection network (MSFSNet) that seamlessly integrates spatial and frequency domain knowledge, selectively recovering richer and more accurate information. Specifically, we initially capture spatial features and input them into dynamic filter selection modules (DFS) at different scales to integrate frequency knowledge. DFS utilizes learnable filters to generate high and low-frequency information and employs a frequency cross-attention mechanism (FCAM) to determine the most information to recover. To learn a multi-scale and accurate set of hybrid features, we develop a skip feature fusion block (SFF) that leverages contextual features to discriminatively determine which information should be propagated in skip-connections. It is worth noting that our DFS and SFF are generic plug-in modules that can be directly employed in existing networks without any adjustments, leading to performance improvements. Extensive experiments across various image restoration tasks demonstrate that our MSFSNet achieves performance that is either superior or comparable to state-of-the-art algorithms.
Northwestern Polytechnical University（西北工业大学）；The University of Adelaide（阿德莱德大学）
In dynamic scenes, images often suffer from dynamic blur due to superposition of motions or low signal-noise ratio resulted from quick shutter speed when avoiding motions. Recovering sharp and clean results from the captured images heavily depends on the ability of restoration methods and the quality of the input. Although existing research on image restoration focuses on developing models for obtaining better restored results, fewer have studied to evaluate how and which input image leads to superior restored quality. In this paper, to better study an image's potential value that can be explo…
Yau Mathematical Sciences Center, Tsinghua University（清华大学丘成桐数学科学中心）；Hisilicon（海思）；Institute for Interdisciplinary Information Science, Tsinghua University（清华大学交叉信息研究院）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（北京雁栖湖应用数学研究院）
Collecting paired training data is difficult in practice, but the unpaired samples broadly exist. Current approaches aim at generating synthesized training data from unpaired samples by exploring the relationship between the corrupted and clean data. This work proposes LUD-VAE, a deep generative method to learn the joint probability density function from data sampled from marginal distributions. Our approach is based on a carefully designed probabilistic graphical model in which the clean and corrupted data domains are conditionally independent. Using variational inference, we maximize the evi…
Dropout the High-rate Downsampling: A Novel Design Paradigm for UHD Image Restoration
University of Science and Technology of China（中国科学技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Shandong Normal University（山东师范大学）；Sun Yat-sen University（中山大学）
Ultra-High-Definition Restoration Transformers with Correlation Matching Transformation
University of California, San Francisco（加利福尼亚大学旧金山分校）；School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computing and Communications, Lancaster University（兰卡斯特大学计算与通信学院）
Neural Discrimination-Prompted Transformers for Efficient UHD Image Restoration and Enhancement
Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）；University of California, San Francisco（加州大学旧金山分校）
Image denoising is still a challenging issue in many computer vision sub-domains. Recent studies show that significant improvements are made possible in a supervised setting. However, few challenges, such as spatial fidelity and cartoon-like smoothing remain unresolved or decisively overlooked. Our study proposes a simple yet efficient architecture for the denoising problem that addresses the aforementioned issues. The proposed architecture revisits the concept of modular concatenation instead of long and deeper cascaded connections, to recover a cleaner approximation of the given image. We fi…
Imaging under photon-scarce situations introduces challenges to many applications as the captured images are with low signal-to-noise ratio and poor luminance. In this paper, we investigate the raw image restoration under low-photon-count conditions by simulating the imaging of quanta image sensor (QIS). We develop a lightweight framework, which consists of a multi-level pyramid denoising network (MPDNet) and a luminance adjustment (LA) module to achieve separate denoising and luminance enhancement. The main component of our framework is the multi-skip attention residual block (MARB), which in…
Unsupervised Part Discovery via Descriptor-Based Masked Image Restoration with Optimized Constraints
Faculty of Engineering and IT, University of Technology Sydney（技术悉尼大学工程与信息学院）；Dept. of Comp. Sci. and Eng., Southern University of Science and Technology（南方科技大学计算机科学与工程系）
DA2Diff: Exploring Degradation-aware Adaptive Diffusion Priors for All-in-One Weather Restoration
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术与产业化协同创新中心）；College of Computer Science and Technology, Anhui University of Technology（安徽工业大学计算机科学与技术学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Exploring Scalable Unified Modeling for General Low-Level Vision
State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学物联网智能城市国家重点实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
Harbin Institute of Technology（哈尔滨工业大学）；Hong Kong Polytechnic University（香港理工大学）
Generative adversarial networks (GANs) have drawn enormous attention due to the simple yet effective training mechanism and superior image generation quality. With the ability to generate photo-realistic high-resolution (e.g., $1024\times1024$) images, recent GAN models have greatly narrowed the gaps between the generated images and the real ones. Therefore, many recent works show emerging interest to take advantage of pre-trained GAN models by exploiting the well-disentangled latent space and the learned GAN priors. In this paper, we briefly review recent progress on leveraging pre-trained la…
As a method of image restoration, image super-resolution has been extensively studied at first. How to transform a low-resolution image to restore its high-resolution image information is a problem that researchers have been exploring. In the early physical transformation methods, the high-resolution pictures generated by these methods always have a serious problem of missing information, and the edges and details can not be well recovered. With the development of hardware technology and mathematics, people begin to use in-depth learning methods for image super-resolution tasks, from direct in…
Automotive Software Innovation Center（汽车软件创新中心）；Institute of Intelligent Software（智能软件研究所）；Saarland University（萨尔大学）；Hangzhou Institute for Advanced Study（杭州高等研究院）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）
AuthFace: Towards Authentic Blind Face Restoration with Face-oriented Generative Diffusion Prior
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；vivo Mobile Communication Co., Ltd（vivo移动通信有限公司）；Nanyang Technological University（南洋理工大学）
School of Robotics, Hunan University（湖南大学机器人学院）；College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）；Technical University Munich（慕尼黑工业大学）
Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network
School of Computer Science, China University of Geosciences（中国地质大学（武汉）计算机学院）；Engineering Research Center of Natural Resource Information Management and Digital Twin Engineering Software, Ministry of Education（教育部自然资源信息管理与数字孪生工程软件工程研究中心）；Second Surveying and Mapping Institute of Hunan Province（湖南省第二测绘院）；Key Laboratory of Natural Resources Monitoring and Supervision in Southern Hilly Region, Ministry of Natural Resources（自然资源部南方山地自然资源监测与监管重点实验室）
IIT Jodhpur（印度理工学院焦特布尔分校）；University of Rouen（鲁昂大学）；Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）
Images with visual and scene text content are ubiquitous in everyday life. However, current image interpretation systems are mostly limited to using only the visual features, neglecting to leverage the scene text content. In this paper, we propose to jointly use scene text and visual channels for robust semantic interpretation of images. We do not only extract and encode visual and scene text cues, but also model their interplay to generate a contextual joint embedding with richer semantics. The contextual embedding thus generated is applied to retrieval and classification tasks on multimedia …
Because multimodal data contains more modal information, multimodal sentiment analysis has become a recent research hotspot. However, redundant information is easily involved in feature fusion after feature extraction, which has a certain impact on the feature representation after fusion. Therefore, in this papaer, we propose a new multimodal sentiment analysis model. In our model, we use BERT + BiLSTM as new feature extractor to capture the long-distance dependencies in sentences and consider the position information of input sequences to obtain richer text features. To remove redundant infor…
Shanghai Jiao Tong University（上海交通大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；RIKEN AIP（理化学研究所先进智能项目中心）；The University of Tokyo（东京大学）；The University of British Columbia（不列颠哥伦比亚大学）；Shanghai AI Laboratory（上海人工智能实验室）
Image restoration algorithms such as super resolution (SR) are indispensable pre-processing modules for object detection in low quality images. Most of these algorithms assume the degradation is fixed and known a priori. However, in practical, either the real degradation or optimal up-sampling ratio rate is unknown or differs from assumption, leading to a deteriorating performance for both the pre-processing module and the consequent high-level task such as object detection. Here, we propose a novel self-supervised framework to detect objects in degraded low resolution images. We utilizes the …
AutoSSVH: Exploring Automated Frame Sampling for Efficient Self-Supervised Video Hashing
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）
Exploring the Landscape of Non-Equilibrium Memories with Neural Cellular Automata
SB Schools, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland（瑞士联邦理工学院（EPFL）SB学院）；Department of Computing and Mathematical Sciences, Caltech, Pasadena, CA 91125, USA（加州理工学院计算与数学科学系）；Institute for Quantum Information and Matter, Caltech, Pasadena, CA 91125, USA（加州理工学院量子信息与物质研究所）；Department of Physics, University of California Berkeley, Berkeley, CA 94720, USA（加州大学伯克利分校物理系）
Recurrent neural networks (RNNs) have shown the ability to improve scene parsing through capturing long-range dependencies among image units. In this paper, we propose dense RNNs for scene labeling by exploring various long-range semantic dependencies among image units. Different from existing RNN based approaches, our dense RNNs are able to capture richer contextual dependencies for each image unit by enabling immediate connections between each pair of image units, which significantly enhances their discriminative power. Besides, to select relevant dependencies and meanwhile to restrain irrel…
Dalian University of Technology（大连理工大学）；Tsinghua University（清华大学）
Glass is very common in the real world. Influenced by the uncertainty about the glass region and the varying complex scenes behind the glass, the existence of glass poses severe challenges to many computer vision tasks, making glass segmentation as an important computer vision task. Glass does not have its own visual appearances but only transmit/reflect the appearances of its surroundings, making it fundamentally different from other common objects. To address such a challenging task, existing methods typically explore and combine useful cues from different levels of features in the deep netw…
Northwest Minzu University（西北民族大学）；Chinese National Information Technology Research Institute（中国民族信息技术研究院）；China Mobile Design Institute Co., Ltd.（中国移动设计院有限公司）
Dunhuang murals are a collection of Chinese style and national style, forming a self-contained Chinese-style Buddhist art. It has very high historical and cultural value and research significance. Among them, the lines of Dunhuang murals are highly general and expressive. It reflects the character's distinctive character and complex inner emotions. Therefore, the outline drawing of murals is of great significance to the research of Dunhuang Culture. The contour generation of Dunhuang murals belongs to image edge detection, which is an important branch of computer vision, aims to extract salien…
