Conditional discrete generative models struggle to faithfully compose multiple input conditions. To address this, we derive a theoretically-grounded formulation for composing discrete probabilistic generative processes, with masked generation (absorbing diffusion) as a special case. Our formulation enables precise specification of novel combinations and numbers of input conditions that lie outside the training data, with concept weighting enabling emphasis or negation of individual conditions. In synergy with the richly compositional learned vocabulary of VQ-VAE and VQ-GAN, our method attains a $63.4\%$ relative reduction in error rate compared to the previous state-of-the-art, averaged across 3 datasets (positional CLEVR, relational CLEVR and FFHQ), simultaneously obtaining an average absolute FID improvement of $-9.58$. Meanwhile, our method offers a $2.3\times$ to $12\times$ real-time speed-up over comparable methods, and is readily applied to an open pre-trained discrete text-to-image model for fine-grained control of text-to-image generation.
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
South China University of Technology(华南理工大学);JD Explore Academy(京东探索研究院);Nanyang Technological University(南洋理工大学);Shanghai Jiao Tong University(上海交通大学)
MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTs
Tsinghua University(清华大学);Shanghai Jiaotong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);VAST(VAST机构);The Chinese University of Hong Kong(香港中文大学)
VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers
Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University(海德堡大学医学院心血管医学人工智能研究所);Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital(海德堡大学医院心脏病学、血管学、肺病学系);DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim(德国心血管研究中心(DZHK)海德堡/曼海姆合作站点);University Medical Center of the Johannes Gutenberg-University Mainz(美因茨约翰内斯·古腾堡大学医学中心)
Controllable Skin Synthesis via Lesion-Focused Vector Autoregression Model
School of Translational Medicine, Faculty of Medicine, Nursing and Health Sciences, Monash University(转化医学学院,医学、护理与健康科学学院,墨尔本大学);Melbourne Sexual Health Centre, Alfred Health(墨尔本性健康中心,阿尔弗雷德健康中心);AIM for Health Lab, Monash University(健康促进实验室,墨尔本大学);Faculty of IT, Monash University(信息技术学院,墨尔本大学);Faculty of Engineering, Monash University(工程学院,墨尔本大学);Faculty of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine(传染病与热带医学学院,伦敦热带医学学院)
本文提出 LF-VAR 模型,通过病灶测量评分与类型标签引导,结合多尺度 VQVAE 和 VAR Transformer,实现基于语言提示的可控高保真皮肤图像合成,FID 平均达 0.74,较 SOTA 提升 6.3%。
Interpolating Discrete Diffusion Models with Controllable Resampling
School of Computation, Information and Technology, Technical University of Munich(计算、信息与技术学院,慕尼黑技术大学);Munich Data Science Institute(慕尼黑数据科学研究所);Munich Center for Machine Learning(慕尼黑机器学习中心)
CCLAP: Controllable Chinese Landscape Painting Generation via Latent Diffusion Model
Beijing Institute of Technology(北京理工大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Institute of Intelligent Computing Technology, Chinese Academy of Sciences(中国科学院智能计算技术研究所);Tomorrow Advancing Life(好未来)
FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Science(中国科学院大学人工智能学院);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室);Wuhan AI Research, Wuhan, China(武汉人工智能研究所)
BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation
Department of Biomedical Data Science, Stanford University School of Medicine(斯坦福大学医学院生物医学数据科学系);Department of Mathematical Modelling, Statistics & Bioinformatics, Ghent University(根特大学数学建模、统计与生物信息学系);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系)
Controllable Diverse Sampling for Diffusion Based Motion Behavior Forecasting
Leibniz University Hannover(汉诺威莱布尼茨大学);Institute of Cartography and Geoinformatics(制图与地理信息研究所);University of Twente(特温特大学);Faculty of Geo-Information Science and Earth Observation(地理信息科学与地球观测学院)
From Mapping to Composing: A Two-Stage Framework for Zero-shot Composed Image Retrieval
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(人工智能与机器人研究所,西安交通大学);Harbin Institute of Technology(哈尔滨工业大学);Ant Group(蚂蚁集团)
Denoise-I2W: Mapping Images to Denoising Words for Accurate Zero-Shot Composed Image Retrieval
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Information Engineering, Minzu University of China(中央民族大学信息工程学院);School of Cyberspace Science and Technology, Beijing Institute of Technology(北京理工大学网络空间科学与技术学院)
LC-NeRF: Local Controllable Face Generation in Neural Randiance Field
Tsinghua University(清华大学);Stanford University(斯坦福大学);Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)