Diffusion models have garnered significant attention since they can effectively learn complex multivariate Gaussian distributions, resulting in diverse, high-quality outcomes. They introduce Gaussian noise into training data and reconstruct the original data iteratively. Central to this iterative process is a single Unet, adapting across time steps to facilitate generation. Recent work revealed the presence of composition and denoising phases in this generation process, raising questions about the Unets' varying roles. Our study dives into the dynamic behavior of Unets within denoising diffusion probabilistic models (DDPM), focusing on (de)convolutional blocks and skip connections across time steps. We propose an analytical method to systematically assess the impact of time steps and core Unet components on the final output. This method eliminates components to study causal relations and investigate their influence on output changes. The main purpose is to understand the temporal dynamics and identify potential shortcuts during inference. Our findings provide valuable insights into the various generation phases during inference and shed light on the Unets' usage patterns across these phases. Leveraging these insights, we identify redundancies in GLIDE (an improved DDPM) and improve inference time by ~27% with minimal degradation in output quality. Our ultimate goal is to guide more informed optimization strategies for inference and influence new model designs.
Lets Play Music: Audio-driven Performance Video Generation
Anhui University(安徽大学);University of Chinese Academy of Sciences(中国科学院大学);Center for Excellence in Brain Science and Intelligence Technology, CAS(中国科学院脑科学与智能技术卓越创新中心)
Time-aware UNet and super-resolution deep residual networks for spatial downscaling
Department of Mathematics and Statistics(数学与统计学系);University of Jyväskylä(于韦斯屈莱大学);Department of Economic Sciences(经济科学系);University of Salento(萨伦托大学);National Centre for HPC, Big Data and Quantum Computing(高性能计算、大数据与量子计算国家中心);National Biodiversity Future Center(国家生物多样性未来中心);University of Helsinki(赫尔辛基大学)
Exploring the design space of diffusion and flow models for data fusion
Northeastern University(东北大学);Western Kentucky University(西肯塔基大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Leibniz-Centre for Agricultural Landscape Research (ZALF)(莱布尼茨农业景观研究中心)
DECADE: A Temporally-Consistent Unsupervised Diffusion Model for Enhanced Rb-82 Dynamic Cardiac PET Image Denoising
Yale University(耶鲁大学);Yale School of Medicine(耶鲁医学院);Siemens Medical Solutions USA, Inc.(西门子医疗解决方案美国公司);Inselspital, Bern University Hospital(伯尔尼大学医院Inselspital);University of Bern(伯尔尼大学);Department of Biomedical Engineering(生物医学工程系);Department of Radiology and Biomedical Imaging(放射科和生物医学成像系);Department of Medicine (Cardiology)(医学系(心脏病学))
JointTuner: Appearance-Motion Adaptive Joint Training for Customized Video Generation
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);Alibaba International Digital Commerce(阿里巴巴国际数字商业)
Denoising diffusion models for high-resolution microscopy image restoration
Frankfurt Institute for Advanced Studies (FIAS)(法兰克福高级研究所);Goethe University Frankfurt(法兰克福大学);Institute of Physical and Theoretical Chemistry(物理与理论化学研究所)
Low-Cost Test-Time Adaptation for Robust Video Editing
UESTC(电子科技大学);USTC(中国科学技术大学);University of Minnesota–Twin Cities(明尼苏达大学-双城分校);Emory University(埃默里大学);Nanjing University(南京大学);Lanzhou University(兰州大学);Ohio State University, Columbus(俄亥俄州立大学-哥伦布分校);Chinese Academy of Sciences(中国科学院);Peking University(北京大学)
VMU-Diff: A Coarse-to-fine Multi-source Data Fusion Framework for Precipitation Nowcasting
Department of Automation, Southeast University(东南大学自动化部门);State Key Laboratory of Virtual Reality Technology and Systems, Beihang University(北京航空航天大学虚拟现实技术与系统国家重点实验室);Key Laboratory of Transportation Meteorology, China Meteorological Administration(中国气象局交通运输气象重点实验室)
Training-Free Condition Video Diffusion Models for single frame Spatial-Semantic Echocardiogram Synthesis
Institute for Artificial Intelligence, University of Engineering and Technology, Vietnam National University(越南国家大学工程技术大学人工智能学院);VinUni-Illinois Smart Health Center, VinUniversity(范南大学校伊利诺伊智能健康中心);College of Engineering & Computer Science, VinUniversity(范南大学工程与计算机科学学院)
Ariadne's Thread of LipSync: Unraveling Forgeries via Inconsistency between Lip Motions and Head Poses
University of Science and Technology of China(中国科学技术大学);Shanghai Jiaotong University(上海交通大学);Peking University(北京大学);Nanyang Technological University(南洋理工大学)
AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation
University of Arkansas(阿肯色大学);University of Maryland(马里兰大学);Carnegie Mellon University(卡内基梅隆大学);University of Science VNU-HCM(胡志明市国家大学自然科学大学);Vietnam National University(越南国家大学)