We propose an approach to enhancing synthetic video realism, which can re-render synthetic videos from a simulator in photorealistic fashion. Our realism enhancement approach is a zero-shot framework that focuses on preserving the multi-level structures from synthetic videos into the enhanced one in both spatial and temporal domains, built upon a diffusion video foundational model without further fine-tuning. Specifically, we incorporate an effective modification to have the generation/denoising process conditioned on estimated structure-aware information from the synthetic video, such as depth maps, semantic maps, and edge maps, by an auxiliary model, rather than extracting the information from a simulator. This guidance ensures that the enhanced videos are consistent with the original synthetic video at both the structural and semantic levels. Our approach is a simple yet general and powerful approach to enhancing synthetic video realism: we show that our approach outperforms existing baselines in structural consistency with the original video while maintaining state-of-the-art photorealism quality in our experiments.
University of Maryland, College Park（马里兰大学学院市分校）；Johns Hopkins University（约翰斯·霍普金斯大学）
In this paper, we introduce a new architecture for few shot learning, the task of teaching a neural network from as few as one or five labeled examples. Inspired by the theoretical results of Alaine et al that Denoising Autoencoders refine features to lie closer to the true data manifold, we present a new training scheme that adds noise at multiple stages of an existing neural architecture while simultaneously learning to be robust to this added noise. This architecture, which we call a Self-Denoising Neural Network (SDNN), can be applied easily to most modern convolutional neural architecture…
Recent denoising algorithms based on the "blind-spot" strategy show impressive blind image denoising performances, without utilizing any external dataset. While the methods excel in recovering highly contaminated images, we observe that such algorithms are often less effective under a low-noise or real noise regime. To address this gap, we propose an alternative denoising strategy that leverages the architectural inductive bias of implicit neural representations (INRs), based on our two findings: (1) INR tends to fit the low-frequency clean image signal faster than the high-frequency noise, an…
Pyramid Self-Contrastive Learning for Single-shot Test-time Ultrasound Image Denoising
Department of Electrical and Computer Engineering, The University of Hong Kong（香港大学电子与计算机工程系）；Department of Biomedical Engineering, Duke University（达特茅斯大学生物医学工程系）
Technische Universität München（慕尼黑工业大学）；Universität Ulm（乌尔姆大学）；The University of Adelaide（阿德莱德大学）
We study the problem of few-shot learning-based denoising where the training set contains just a handful of clean and noisy samples. A solution to mitigate the small training set issue is to pre-train a denoising model with small training sets containing pairs of clean and synthesized noisy signals, produced from empirical noise priors, and fine-tune on the available small training set. While such transfer learning seems effective, it may not generalize well because of the limited amount of training data. In this work, we propose a new meta-learning training approach for few-shot learning-base…
Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or mani…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）
Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the Denoising Diffusion Null-Space Model (DDNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs a pre-trained off-the-shelf diffusion model as the generative prior, without any extra training or network modifications. By refining only the null-space contents during the reverse diffusion process, we c…
DeltaDeno: Zero-Shot Anomaly Generation via Delta-Denoising Attribution
Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院）；CASI Vision Technology CO., LTD., Luoyang, China（CASI视觉科技有限公司）；School of Artificial Intelligence and Robotics, Hunan University, Changsha, China（湖南大学人工智能与机器人学院）
Zero-Shot Image Denoising for High-Resolution Electron Microscopy
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Lingang Laboratory（临港实验室）；School of Physical Science and Technology, ShanghaiTech University（上海科技大学物理科学与技术学院）；School of Software, Tsinghua University（清华大学软件学院）；School of Biomedical Engineering and Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学生物医学工程学院与医学机器人研究所）
RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models
Peking University（北京大学）；University of Science and Technology of China（中国科学技术大学）；Southeast University（东南大学）；Stanford University（斯坦福大学）；Tsinghua University（清华大学）
Expert Validation of Synthetic Cervical Spine Radiographs Generated with a Denoising Diffusion Probabilistic Model
Cumming School of Medicine, University of Calgary（卡里尔大学医学学院）；Division of Neurosurgery, Department of Clinical Neurosciences, University of Calgary（卡里尔大学临床神经科学系神经外科部）；Department of Radiology, University of Calgary（卡里尔大学放射科）；Division of Neurosurgery, Department of Surgery, University of Toronto（多伦多大学外科系神经外科部）；Department of Medical Imaging, University of Toronto（多伦多大学医学影像科）；Project neuroArm, Department of Clinical Sciences, University of Calgary（卡里尔大学临床科学部neuroArm项目）
University Paris-Est（巴黎东大学）；Ecole des Ponts ParisTech（巴黎高科桥路学院）
Given an initial recognition model already trained on a set of base classes, the goal of this work is to develop a meta-model for few-shot learning. The meta-model, given as input some novel classes with few training examples per class, must properly adapt the existing recognition model into a new model that can correctly classify in a unified way both the novel and the base classes. To accomplish this goal it must learn to output the appropriate classification weight vectors for those two types of classes. To build our meta-model we make use of two main innovations: we propose the use of a De…
Denoise-I2W: Mapping Images to Denoising Words for Accurate Zero-Shot Composed Image Retrieval
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Information Engineering, Minzu University of China（中央民族大学信息工程学院）；School of Cyberspace Science and Technology, Beijing Institute of Technology（北京理工大学网络空间科学与技术学院）
A Zero-Shot Deep Image Prior Framework for Denoising and Deconvolution in Fluorescence Microscopy
Wenzhou Key Laboratory of Biomedical Imaging（温州生物医学成像重点实验室）；Zhejiang Key Laboratory of Soft Matter Biomedical Materials（浙江软物质生物医学材料重点实验室）；Wenzhou Institute, University of Chinese Academy of Sciences（中国科学院大学温州研究院）；Postgraduate Training Base Alliance of Wenzhou Medical University（温州医学院研究生联合培养基地联盟）
Filter2Noise: A Framework for Interpretable and Zero-Shot Low-Dose CT Image Denoising
Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nuremberg（模式识别实验室，弗里德里希-亚历山大大学埃尔朗根-纽伦堡）；Siemens Healthineers AG（西门子医疗影像有限公司）；Department of Radiology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（放射科，复杂严重罕见疾病国家重点实验室，北京协和医院，中国医学科学院及北京协和医学院）；Theranostics and Translational Research Center, National Infrastructures for Translational Medicine, Institute of Clinical Medicine, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（诊疗与转化研究中心，国家转化医学基础设施，临床医学研究院，北京协和医院，中国医学科学院及北京协和医学院）
"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models
University of California, Santa Cruz（加州大学圣克ruz分校）；NVIDIA Research（NVIDIA研究）；Northeastern University（东北大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Median2Median: Zero-shot Suppression of Structured Noise in Images
Biomedical Imaging Center, Center for Computational Innovations, Center for Biotechnology & Interdisciplinary Studies, Department of Biomedical Engineering, School of Engineering, Rensselaer Polytechnic Institute（生物医学成像中心、计算创新中心、生物技术与跨学科研究所以及生物医学工程系、工程学院、罗切斯特理工学院）
As synthetic imagery is used more frequently in training deep models, it is important to understand how different synthesis techniques impact the performance of such models. In this work, we perform a thorough evaluation of the effectiveness of several different synthesis techniques and their impact on the complexity of classifier domain adaptation to the "real" underlying data distribution that they seek to replicate. In addition, we propose a novel learned synthesis technique to better train classifier models than state-of-the-art offline graphical methods, while using significantly less com…
Lunenfeld-Tanenbaum Research Institute（卢嫩菲尔德-塔南鲍姆研究所）；University of Toronto（多伦多大学）
Because noise can interfere with downstream analysis, image denoising has come to occupy an important place in the image processing toolbox. The most accurate state-of-the-art denoisers typically train on a representative dataset. But gathering a training set is not always feasible, so interest has grown in blind zero-shot denoisers that train only on the image they are denoising. The most accurate blind-zero shot methods are blind-spot networks, which mask pixels and attempt to infer them from their surroundings. Other methods exist where all neurons participate in forward inference, however …
ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing
Tsinghua University（清华大学）；Nanjing University（南京大学）；Kling Team, Kuaishou Technology（快手科技影团队）；ShanghaiTech University（上海科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
