Phase retrieval refers to the problem of recovering an image from the magnitudes of its complex-valued linear measurements. Since the problem is ill-posed, the recovery requires prior knowledge on the unknown image. We present DOLPH as a new deep model-based architecture for phase retrieval that integrates an image prior specified using a diffusion model with a nonconvex data-fidelity term for phase retrieval. Diffusion models are a recent class of deep generative models that are relatively easy to train due to their implementation as image denoisers. DOLPH reconstructs high-quality solutions by alternating data-consistency updates with the sampling step of a diffusion model. Our numerical results show the robustness of DOLPH to noise and its ability to generate several candidate solutions given a set of measurements.
I2I-PR: Deep Iterative Refinement for Phase Retrieval using Image-to-Image Diffusion Models
Department of Electrical Engineering, Middle East Technical University (METU)（电子工程系，中东技术大学）；Department of Applied Mathematics and Computer Science, Technical University of Denmark（应用数学与计算机科学系，丹麦技术大学）
Ludwig-Maximilian University Munich（慕尼黑路德维希-马克西米利安大学）
Novel architectures have recently improved generative image synthesis leading to excellent visual quality in various tasks. Of particular note is the field of ``AI-Art'', which has seen unprecedented growth with the emergence of powerful multimodal models such as CLIP. By combining speech and image synthesis models, so-called ``prompt-engineering'' has become established, in which carefully selected and composed sentences are used to achieve a certain visual style in the synthesized image. In this note, we present an alternative approach based on retrieval-augmented diffusion models (RDMs). In…
Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
King Abdullah University of Science and Technology（国王阿卜杜勒-阿齐兹大学科学与技术大学）；Visual Recognition Group（视觉识别组）；Faculty of Electrical Engineering（电气工程学院）；Czech Technical University in Prague（布拉格捷克技术大学）
DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Technology, Nanjing University（南京大学计算机科学与技术系）
Recent text-to-image models have achieved impressive results. However, since they require large-scale datasets of text-image pairs, it is impractical to train them on new domains where data is scarce or not labeled. In this work, we propose using large-scale retrieval methods, in particular, efficient k-Nearest-Neighbors (kNN), which offers novel capabilities: (1) training a substantially small and efficient text-to-image diffusion model without any text, (2) generating out-of-distribution images by simply swapping the retrieval database at inference time, and (3) performing text-driven local …
Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval
Queen Mary University of London（伦敦玛丽女王大学）；Sony AI（索尼人工智能公司）；Adobe Research（奥多比研究院）；WICT, Peking University（北京大学信息科学技术学院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）
MOST: Motion Diffusion Model for Rare Text via Temporal Clip Banzhaf Interaction
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；Department of Computer Science, University of Durham（计算机科学系，达勒姆大学）
Conditional generative models such as DALL-E and Stable Diffusion generate images based on a user-defined text, the prompt. Finding and refining prompts that produce a desired image has become the art of prompt engineering. Generative models do not provide a built-in retrieval model for a user's information need expressed through prompts. In light of an extensive literature review, we reframe prompt engineering for generative models as interactive text-based retrieval on a novel kind of "infinite index". We apply these insights for the first time in a case study on image generation for game de…
WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models
Luleå University of Technology（吕勒奥理工大学）；National Technical University of Athens（雅典国立技术大学）；Friedrich-Alexander-Universität（弗里德里希-亚历山大大学）；University of West Attica（西阿提卡大学）；University of Ioannina（约阿尼纳大学）
University of Maryland College Park（马里兰大学帕克分校）；New York University（纽约大学）
Cutting-edge diffusion models produce images with high quality and customizability, enabling them to be used for commercial art and graphic design purposes. But do diffusion models create unique works of art, or are they replicating content directly from their training sets? In this work, we study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated. Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as trai…
Solving Inverse Problems using Diffusion with Iterative Colored Renoising
Dept. Electrical and Computer Engineering, The Ohio State University（电气与计算机工程系，俄亥俄州立大学）；Dept. Biomedical Engineering, The Ohio State University（生物医学工程系，俄亥俄州立大学）
