In this work, we observe that model trained on vast general images via masking strategy, has been naturally embedded with their distribution knowledge, thus spontaneously attains the underlying potential for strong image denoising. Based on this observation, we propose a novel zero-shot denoising paradigm, i.e., Masked Pre-train then Iterative fill (MPI). MPI first trains model via masking and then employs pre-trained weight for high-quality zero-shot image denoising on a single noisy image. Concretely, MPI comprises two key procedures: 1) Masked Pre-training involves training model to reconstruct massive natural images with random masking for generalizable representations, gathering the potential for valid zero-shot denoising on images with varying noise degradation and even in distinct image types. 2) Iterative filling exploits pre-trained knowledge for effective zero-shot denoising. It iteratively optimizes the image by leveraging pre-trained weights, focusing on alternate reconstruction of different image parts, and gradually assembles fully denoised image within limited number of iterations. Comprehensive experiments across various noisy scenarios underscore the notable advances of MPI over previous approaches with a marked reduction in inference time. Code available at https://github.com/krennic999/MPI.
Lunenfeld-Tanenbaum Research Institute（卢嫩菲尔德-塔南鲍姆研究所）；University of Toronto（多伦多大学）
Because noise can interfere with downstream analysis, image denoising has come to occupy an important place in the image processing toolbox. The most accurate state-of-the-art denoisers typically train on a representative dataset. But gathering a training set is not always feasible, so interest has grown in blind zero-shot denoisers that train only on the image they are denoising. The most accurate blind-zero shot methods are blind-spot networks, which mask pixels and attempt to infer them from their surroundings. Other methods exist where all neurons participate in forward inference, however …
You KAN Do It in a Single Shot: Plug-and-Play Methods with Single-Instance Priors
Department of Applied Mathematics and Theoretical Physics（应用数学与理论物理系）；University of Cambridge（剑桥大学）；Yau Mathematical Sciences Center（叶中伟数学科学中心）；Tsinghua University（清华大学）
提出 KAN-PnP 优化框架，将 KAN 作为去噪器融入即插即用范式解决单实例先验逆问题，证明了其 Lipschitz 连续性与收敛性，并在超分辨率等任务上以更少迭代实现高精度。
MPISuperRes-PnP: A Super-Resolution Zero-Shot Plug-and-Play Reconstruction Algorithm for Magnetic Particle Imaging
Institute for the Protection of Terrestrial Infrastructures, German Aerospace Center (DLR)（德国航空航天中心陆地基础设施保护研究所）；Hochschule Darmstadt（达姆施塔特应用科学大学）；Data Science Institute, European University of Technology（欧洲技术大学数据科学研究所）；Technische Hochschule Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用技术大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
Self-supervised blind denoising for Poisson-Gaussian noise remains a challenging task. Pseudo-supervised pairs constructed from single noisy images re-corrupt the signal and degrade the performance. The visible blindspots solve the information loss in masked inputs. However, without explicitly noise sensing, mean square error as an objective function cannot adjust denoising intensities for dynamic noise levels, leading to noticeable residual noise. In this paper, we propose Blind2Sound, a simple yet effective approach to overcome residual noise in denoised images. The proposed adaptive re-visi…
Single-Shot Plug-and-Play Methods for Inverse Problems
University of Cambridge（剑桥大学）；Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；Lingnan University（岭南大学）；Tsinghua University（清华大学）；City University of Hong Kong（香港城市大学）
Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究院）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
Convergence Research Institute, Sungkyunkwan University（convergence research institute, 首尔大学）；School of Information Technology, Deakin University（信息科技学院, 德金大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系, 首尔大学）
Deep learning has thrived by training on large-scale datasets. However, in robotics applications sample efficiency is critical. We propose a novel adaptive masked proxies method that constructs the final segmentation layer weights from few labelled samples. It utilizes multi-resolution average pooling on base embeddings masked with the label to act as a positive proxy for the new class, while fusing it with the previously learned class signatures. Our method is evaluated on PASCAL-$5^i$ dataset and outperforms the state-of-the-art in the few-shot semantic segmentation. Unlike previous methods,…
Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance
CMAP, Ecole Polytechnique（高等理工学院CMAP）；Institute of Foundation Models（基础模型研究所）；MBZUAI（穆扎夫技术研究院）；EPITA；Sorbonne University（索邦大学）；Lagrange Mathematics and Computing Research Center（拉格朗日数学与计算研究中心）；EPITA Research Lab（EPITA研究实验室）；KTH Royal Institute of Technology（皇家理工学院）
State-of-the-art computer vision models are mostly trained with supervised learning using human-labeled images, which limits their scalability due to the expensive annotation cost. While self-supervised representation learning has achieved impressive progress, it still requires a second stage of finetuning on labeled data. On the other hand, models pre-trained with large-scale text-image supervision (e.g., CLIP) have enabled zero-shot transfer to downstream image classification tasks. However, the zero-shot performance of CLIP-like models are often insufficient for real-world adoption. In this…
XJTLU（西交利物浦大学）；University of Liverpool（利物浦大学）；Dinnar Automation Technology（迪纳自动化技术公司）
Few-shot segmentation has been attracting a lot of attention due to its effectiveness to segment unseen object classes with a few annotated samples. Most existing approaches use masked Global Average Pooling (GAP) to encode an annotated support image to a feature vector to facilitate query image segmentation. However, this pipeline unavoidably loses some discriminative information due to the average operation. In this paper, we propose a simple but effective self-guided learning approach, where the lost critical information is mined. Specifically, through making an initial prediction for the a…
MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院保罗·C·劳特伯生物医学成像研究中心）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
The University of Hong Kong（香港大学）；ARC Lab（ARC实验室）；UC Berkeley（加州大学伯克利分校）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；Tencent PCG（腾讯平台与内容事业群）
Dominant pre-training work for video-text retrieval mainly adopt the "dual-encoder" architectures to enable efficient retrieval, where two separate encoders are used to contrast global video and text representations, but ignore detailed local semantics. The recent success of image BERT pre-training with masked visual modeling that promotes the learning of local visual context, motivates a possible solution to address the above limitation. In this work, we for the first time investigate masked visual modeling in video-text pre-training with the "dual-encoder" architecture. We perform Masked vis…
Huazhong University of Science & Technology（华中科技大学）；Tencent PCG（腾讯平台与内容事业群）；Foundation Technology Center（基础技术中心）
Both masked image modeling (MIM) and natural language supervision have facilitated the progress of transferable visual pre-training. In this work, we seek the synergy between two paradigms and study the emerging properties when MIM meets natural language supervision. To this end, we present a novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations, encoded by the text encoder, serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets. The…
Texas A&M University（得克萨斯农工大学）；Microsoft（微软）；University of Texas at Austin（得克萨斯大学奥斯汀分校）
Recently, both Contrastive Learning (CL) and Mask Image Modeling (MIM) demonstrate that self-supervision is powerful to learn good representations. However, naively combining them is far from success. In this paper, we start by making the empirical observation that a naive joint optimization of CL and MIM losses leads to conflicting gradient directions - more severe as the layers go deeper. This motivates us to shift the paradigm from combining loss at the end, to choosing the proper learning method per network layer. Inspired by experimental observations, we find that MIM and CL are suitable …
University of Southern California（南加州大学）；Microsoft Corporation（微软公司）
Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning. However, these VL models are hard to deploy for real-world applications due to their impractically huge sizes and slow inference speed. To solve this limitation, we study prompt-based low-resource learning of VL tasks with our proposed method, FewVLM, relatively smaller than recent few-shot learners. For FewVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM). Furthe…
MMCLIP: Cross-modal Attention Masked Modelling for Medical Language-Image Pre-Training
University of Technology Sydney（悉尼科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The Australian National University（澳大利亚国立大学）；University of Adelaide（阿德莱德大学）
Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning
Paul C. Lauterbur Research Center for Biomedical Imaging（保罗·C·劳特伯生物医学成像研究中心）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Pengcheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Harvard University（哈佛大学）；Shanghai AI Lab（上海人工智能实验室）
