Joint machine learning models that allow synthesizing and classifying data often offer uneven performance between those tasks or are unstable to train. In this work, we depart from a set of empirical observations that indicate the usefulness of internal representations built by contemporary deep diffusion-based generative models not only for generating but also predicting. We then propose to extend the vanilla diffusion model with a classifier that allows for stable joint end-to-end training with shared parameterization between those objectives. The resulting joint diffusion model outperforms recent state-of-the-art hybrid methods in terms of both classification and generation quality on all evaluated benchmarks. On top of our joint training approach, we present how we can directly benefit from shared generative and discriminative representations by introducing a method for visual counterfactual explanations.
Generating visual layouts is an essential ingredient of graphic design. The ability to condition layout generation on a partial subset of component attributes is critical to real-world applications that involve user interaction. Recently, diffusion models have demonstrated high-quality generative performances in various domains. However, it is unclear how to apply diffusion models to the natural representation of layouts which consists of a mix of discrete (class) and continuous (location, size) attributes. To address the conditioning layout generation problem, we introduce DLT, a joint discre…
Renmin University of China（中国人民大学）；Peking University（北京大学）；Microsoft Research（微软研究院）
We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupled denoising autoencoders. In contrast to existing single-modal diffusion models, MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Two subnets for audio and video learn to gradually generate aligned audio-video pairs from Gaussian noises. To ensure semantic con…
Projected Coupled Diffusion for Test-Time Constrained Joint Generation
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Faculty of Computer Science and Information Technology, Universiti Malaya（马来亚大学计算机科学与信息技术学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）
Couple to Control: Joint Initial Noise Design in Diffusion Models
Department of Computer Science（计算机科学系）；Rutgers University（罗格斯大学）；Department of EECS（电子工程与计算机科学系）；University of Michigan（密歇根大学）；Department of Statistics（统计学系）
Product-of-Gaussian-Mixture Diffusion Models for Joint Nonlinear MRI Reconstruction
Graz University of Technology（格拉茨技术大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Biomedical Imaging Group and Center for Biomedical Imaging（生物医学成像组和生物医学成像中心）
Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Westlake University（西交大学）；Zhejiang University（浙江大学）；Monash University（墨尔本大学）
Overcoming Dimensional Factorization Limits in Discrete Diffusion Models through Quantum Joint Distribution Learning
Faculty of Innovation Engineering, Macau University of Science and Technology（澳门科学技术大学创新工程学院）；Department of Electrical and Computer Engineering, New Jersey Institute of Technology（新泽西理工学院电气与计算机工程系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Electronic and Information Engineering, Foshan University（佛山大学电子与信息工程学院）；College of Mathematics and Informatics, South China Agricultural University（华南农业大学数学与信息学院）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Microsoft Research（微软研究院）
Language-guided image generation has achieved great success nowadays by using diffusion models. However, texts can be less detailed to describe highly-specific subjects such as a particular dog or a certain car, which makes pure text-to-image generation not accurate enough to satisfy user requirements. In this work, we present a novel Unified Multi-Modal Latent Diffusion (UMM-Diffusion) which takes joint texts and images containing specified subjects as input sequences and generates customized images with the subjects. To be more specific, both input texts and images are encoded into one unifi…
GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
Codec Avatars Lab, Meta（Meta编码器动画实验室）；University of Tübingen（图宾根大学）；Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息学院，萨尔兰信息校园）
Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning
Korea Institute for Advanced Study（韩国高等研究院）；University of Seoul（首尔市立大学）；Seoul National University（首尔国立大学）；Hanyang University（汉阳大学）；Saige Research（赛奇研究院）
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）；Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱比锡预防研究与流行病学研究所 – BIPS）；Faculty of Mathematics and Computer Science, University of Bremen（不莱梅大学数学与计算机科学学院）；Faculty of Physics and Electrical Engineering, University of Bremen（不莱梅大学物理与电气工程学院）
Inria（法国国家信息与自动化技术研究院）；PSL Research University（巴黎社会科学高等研究院）；CMAP（巴黎高等理工学院应用数学与计算科学实验室）；Ecole Polytechnique Palaiseau, France（法国巴黎高等理工学院Palaiseau分校）；Sakana AI Tokyo, Japan（日本东京Sakana AI公司）
Diffusion-Based Joint Temperature and Precipitation Emulation of Earth System Models
Western Washington University（西华盛顿大学）；Utah State University（犹他州立大学）；Joint Global Change Research Institute（全球变化联合研究所）；Pacific Northwest National Laboratory（太平洋西北国家实验室）；AI and Data Analytics Division, Pacific Northwest National Laboratory（太平洋西北国家实验室数据分析与人工智能分部）
MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
Music and Audio Research Group (MARG)（音乐与音频研究组）；Interdisciplinary Program in Artificial Intelligence (IPAI)（人工智能跨学科项目）；AIIS（人工智能研究所）；Department of Intelligence and Information（智能与信息系）
DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Technology, Nanjing University（南京大学计算机科学与技术系）
DualDiT: A Conditional Dual-Output Diffusion Transformer for Joint OCT Image and Segmentation Mask Generation
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano (HUMAN-tech), Universitat Politècnica de València（瓦伦西亚理工大学以人类为中心技术大学研究所）；Artikode Intelligence S.L（Artikode智能公司）；Biomedical Technology Center of the Medical Faculty, University of Muenster（明斯特大学医学院生物医学技术中心）；Department of Ophthalmology, University of Muenster Medical Centre（明斯特大学医学中心眼科）
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
Hong Kong University of Science and Technology（香港科学与技术大学）；Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Independent Researcher（独立研究者）
