Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. These results demonstrate that MLLMs provide strong priors for denoising visual representations and that, by conditioning on evolving noisy representations, test-time compute can be productively spent on repeated MLLM conditioning in modern T2I systems.
Image synthesis under multi-modal priors is a useful and challenging task that has received increasing attention in recent years. A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities (i.e. priors) and corresponding outputs. In recent work, a variational auto-encoder (VAE) model was trained in a weakly supervised manner to address this challenge. Since the generative power of VAEs is usually limited, it is difficult for this method to synthesize images belonging to complex distributions. To this end, we propose a solution base…
A Unified Framework for Multimodal Image Reconstruction and Synthesis using Denoising Diffusion Models
Department of Computer Science and Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校计算机科学与工程系）；Mallinckrodt Institute of Radiology, Washington University in St. Louis（华盛顿大学圣路易斯分校马林克罗德特放射医学研究所）；Department of Electrical and Systems Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校电气与系统工程系）；Department of Neurology, Washington University in St. Louis（华盛顿大学圣路易斯分校神经病学系）；Department of Biomedical Engineering, Washington University in St. Louis（华盛顿大学圣路易斯分校生物医学工程系）；Division of Biology and Biomedical Sciences, Washington University in St. Louis（华盛顿大学圣路易斯分校生物学与生物医学科学 division）；Department of Electrical and Computer Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校电气与计算机工程系）
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence（佛罗伦萨大学媒体集成与传播中心（MICC））；University of Florence（佛罗伦萨大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of Pisa（比萨大学）
ChangeBridge: Spatiotemporal Image Generation with Multimodal Controls for Remote Sensing
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Zhongguancun Academy（中关村学院）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学新领域创成科学研究科）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）
We formulate monocular depth estimation using denoising diffusion models, inspired by their recent successes in high fidelity image generation. To that end, we introduce innovations to address problems arising due to noisy, incomplete depth maps in training data, including step-unrolled denoising diffusion, an $L_1$ loss, and depth infilling during training. To cope with the limited availability of data for supervised training, we leverage pre-training on self-supervised image-to-image translation tasks. Despite the simplicity of the approach, with a generic loss and architecture, our DepthGen…
UniRect-CoT: Enhancing Generation in Unified Multimodal Models via Reflective Rectification with Inherent Understanding
School of Software Technology, Zhejiang University（浙江大学软件技术学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Computer Science（计算机科学学院）
Inconsistency-aware Multimodal Schrödinger Bridge for Deepfake Localization
Department of Computer Science and Techonology, Huaqiao University（华侨大学计算机科学与技术系）；Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Huaqiao University（厦门计算机视觉与模式识别重点实验室）；Tongji University（同济大学）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）
In this paper, we propose a new multimodal image denoising approach to attenuate white Gaussian additive noise in a given image modality under the aid of a guidance image modality. The proposed coupled image denoising approach consists of two stages: coupled sparse coding and reconstruction. The first stage performs joint sparse transform for multimodal images with respect to a group of learned coupled dictionaries, followed by a shrinkage operation on the sparse representations. Then, in the second stage, the shrunken representations, together with coupled dictionaries, contribute to the reco…
Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University（IRMV实验室，自动化系，上海交通大学）；Meta Reality Labs（Meta现实实验室）；the Department of Electronic Engineering, Shanghai Jiao Tong University（电子工程系，上海交通大学）；the School of Information and Control Engineering, China University of Mining and Technology（信息与控制工程学院，中国矿业大学）；the College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）
Cycle-guided Denoising Diffusion Probability Model for 3D Cross-modality MRI Synthesis
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models
Jinan University, Guangzhou, China；Department of Rehabilitation Medicine, The First Affiliated Hospital of Jinan University, Guangzhou, China；Western University, Ontario, Canada；Guangdong University of Technology, Guangzhou, China；Tsinghua University, Beijing, China
MID: A Self-supervised Multimodal Iterative Denoising Framework
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（自动化与智能感知学院，上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理重点实验室，中华人民共和国教育部）
Modeling Pedestrian Intrinsic Uncertainty for Multimodal Stochastic Trajectory Prediction via Energy Plan Denoising
School of Computing, Macquarie University（麦考瑞大学计算学院）；Data 61, CSIRO（联邦科学与工业研究组织Data 61研究院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）
Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification
The Guangdong Provincial Key Laboratory of Computational Intelligence and Cyberspace Information, the School of Computer Science and Engineering, South China University of Technology（广东省计算智能与网络信息重点实验室、计算机科学与工程学院、华南理工大学）；The Pazhou Laboratory（琶洲实验室）
UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
Stanford University（斯坦福大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Toronto（多伦多大学）；University of Oxford（牛津大学）；UC, Riverside（加州大学河滨分校）；MatrAIx；University of Notre Dame（圣母大学）；Carnegie Mellon University（卡内基梅隆大学）；UNC–Chapel Hill（北卡罗来纳大学教堂山分校）
PMMD: A pose-guided multi-view multi-modal diffusion for person generation
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；City University of Hong Kong, Hong Kong, China（香港城市大学）；Peking University, Beijing, China（北京大学）；Tsinghua University, Beijing, China（清华大学）
BDHT: Generative AI Enables Causality Analysis for Mild Cognitive Impairment
Hubei University of Economics（湖北经济学院）；Hong Kong Baptist University（香港浸会大学）；Shenzhen University（深圳大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning
Sydney AI Centre, The University of Sydney（悉尼大学悉尼人工智能中心）；Institute of Image Communication and Network Engineering, Shanghai JiaoTong University（上海交通大学图像通信与网络工程研究所）
Optical coherence tomography (OCT) is a prevalent imaging technique for retina. However, it is affected by multiplicative speckle noise that can degrade the visibility of essential anatomical structures, including blood vessels and tissue layers. Although averaging repeated B-scan frames can significantly improve the signal-to-noise-ratio (SNR), this requires longer acquisition time, which can introduce motion artifacts and cause discomfort to patients. In this study, we propose a learning-based method that exploits information from the single-frame noisy B-scan and a pseudo-modality that is c…
