We demonstrate text as a strong cross-modal interface. Rather than relying on deep embeddings to connect image and language as the interface representation, our approach represents an image as text, from which we enjoy the interpretability and flexibility inherent to natural language. We employ an autoencoder that uses a pre-trained text-to-image diffusion model for decoding. The encoder is trained to transform an input image into text, which is then fed into the fixed text-to-image diffusion decoder to reconstruct the original input -- a process we term De-Diffusion. Experiments validate both the precision and comprehensiveness of De-Diffusion text representing images, such that it can be readily ingested by off-the-shelf text-to-image tools and LLMs for diverse multi-modal tasks. For example, a single De-Diffusion model can generalize to provide transferable prompts for different text-to-image tools, and also achieves a new state of the art on open-ended vision-language tasks by simply prompting large language models with few-shot examples.
UniDiff: Parameter-Efficient Adaptation of Diffusion Models for Land Cover Classification with Multi-Modal Remotely Sensed Imagery and Sparse Annotations
Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation
East China University of Science and Technology(华东理工大学);vivo Mobile Communication Co., Ltd(维沃移动通信有限公司);Peng Cheng Laboratory(鹏城实验室);Singapore Management University(新加坡管理大学)
FlexiD-Fuse: Flexible number of inputs multi-modal medical image fusion based on diffusion model
School of Physics(物理学院);Optoelectronic Engineering, Foshan University(光电工程学院,佛山大学);Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology(粤港澳联合智能微纳光电技术实验室);Guangdong Provincial Key Laboratory of Industrial Intelligent Inspection Technology(广东省工业智能检测技术重点实验室);School of Information Engineering(信息工程学院);Automation, Kunming University of Science(自动化系,昆明理工大学)
VI-Diff: Unpaired Visible-Infrared Translation Diffusion Model for Single Modality Labeled Visible-Infrared Person Re-identification
University of Chinese Academy of Sciences (UCAS)(中国科学院大学);The Center for Research on Intelligent Perception and Computing (CRIPAC)(智能感知与计算研究中心);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所)
Wavelet-Fusion Diffusion Model for Multimodal Brain MRI Synthesis with Modality and Metadata Conditioning
Department of Psychology & Neuroscience, Auckland University of Technology(心理学与神经科学系,奥克兰技术大学);Department of Psychiatry, University of Melbourne(精神病学系,墨尔本大学)
Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Nanjing University(南京大学);The University of Sydney(悉尼大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学)
MMGen: Unified Multi-modal Image Generation and Understanding in One Go
The University of Hong Kong(香港大学);The University of Sydney(悉尼大学);AIsphere;Tsinghua University(清华大学);Hong Kong University of Science and Technology(香港科技大学);Texas A&M University(德克萨斯农工大学)
CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusion
University of Freiburg(弗赖堡大学);Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Technical University of Munich(慕尼黑技术大学);Huawei Heisenberg Research Center (Munich)(华为海森堡研究中心)
Text Modality Oriented Image Feature Extraction for Detecting Diffusion-based DeepFake
East China Normal University(华东师范大学);Nanyang Technological University(南洋理工大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);New York University(纽约大学);Wuhan University(武汉大学)
Diffusion based Zero-shot Medical Image-to-Image Translation for Cross Modality Segmentation
Athinoula A. Martinos Center for Biomedical Imaging(阿西诺拉·A.马蒂诺斯生物医学成像中心);Massachusetts General Hospital(麻省总医院);Harvard University(哈佛大学);Inria centre at Université Côte d’Azur(蔚蓝海岸大学国家信息与自动化研究所中心);Temple University(天普大学);Georg-August-University of Göttingen(哥廷根大学)
MCAD: Multi-modal Conditioned Adversarial Diffusion Model for High-Quality PET Image Reconstruction
School of Computer Science, Sichuan University(四川大学计算机学院);The Hong Kong Polytechnic University(香港理工大学);School of Computer Science, Chengdu University of Information Technology(成都信息工程大学计算机学院)
A Survey on Adversarial Attacks and Defenses for Diffusion Models Across Multiple Modalities
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Stanford University(斯坦福大学);Georgia Institute of Technology(佐治亚理工学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
Cross-conditioned Diffusion Model for Medical Image to Image Translation
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Xi’an Jiaotong University(西安交通大学);The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学)
PMMD: A pose-guided multi-view multi-modal diffusion for person generation
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));City University of Hong Kong, Hong Kong, China(香港城市大学);Peking University, Beijing, China(北京大学);Tsinghua University, Beijing, China(清华大学)
Cross-modality image synthesis from TOF-MRA to CTA using diffusion-based models
Charité – Universitätsmedizin Berlin(柏林夏里特医学院);Freie Universität Berlin(柏林自由大学);Humboldt-Universität zu Berlin(柏林洪堡大学);Mie University Graduate School of Medicine(三重大学医学院);Mie Chuo Medical Center(三重中央医疗中心)
MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZIU-UIUC Institute, Zhejiang University(浙江大学ZIU-UIUC研究院);School of Public Health, Zhejiang University(浙江大学公共卫生学院)