Recently proposed generative models for discrete data, such as Masked Diffusion Models (MDMs), exploit conditional independence approximations to reduce the computational cost of popular Auto-Regressive Models (ARMs), at the price of some bias in the sampling distribution. We study the resulting computation-vs-accuracy trade-off, providing general error bounds (in relative entropy) that depend only on the average number of tokens generated per iteration and are independent of the data dimensionality (i.e. sequence length), thus supporting the empirical success of MDMs. We then investigate the gain obtained by using non-constant schedule sizes (i.e. varying the number of unmasked tokens during the generation process) and identify the optimal schedule as a function of a so-called information profile of the data distribution, thus allowing for a principled optimization of schedule sizes. We define methods directly as sampling algorithms and do not use classical derivations as time-reversed diffusion processes, leading us to simple and transparent proofs.
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心,教育部);Huawei Noah’s Ark Lab(华为诺亚实验室)
Inria(法国国家信息与自动化技术研究院);PSL Research University(巴黎社会科学高等研究院);CMAP(巴黎高等理工学院应用数学与计算科学实验室);Ecole Polytechnique Palaiseau, France(法国巴黎高等理工学院Palaiseau分校);Sakana AI Tokyo, Japan(日本东京Sakana AI公司)
CDMT-EHR: A Continuous-Time Diffusion Framework for Generating Mixed-Type Time-Series Electronic Health Records
Department of Core Informatics, Graduate School of Informatics, Osaka Metropolitan University(核心信息学系,信息学研究生学校,大阪 Metropolitan 大学);Smart Data & Knowledge Services Department, German Research Center for Artificial Intelligence (DFKI GmbH)(智能数据与知识服务部,德国人工智能研究中心(DFKI GmbH));Department of Hematology and Department of Laboratory Medicine and Medical Informatics, Graduate School of Medicine, Osaka Metropolitan University(血液科和实验室医学与医学信息学系,医学研究生学校,大阪 Metropolitan 大学)
MODA: A Unified 3D Diffusion Framework for Multi-Task Target-Aware Molecular Generation
School of Artificial Intelligence, Shenzhen University, Shenzhen, China(深圳大学人工智能学院);School of Computer Science, University of Nottingham Ningbo, Ningbo, China(诺丁汉大学宁波校区计算机科学学院);Biology Department, University of California San Diego, California, USA(加州大学圣地亚哥分校生物系)
Monte Carlo Tree Diffusion with Multiple Experts for Protein Design
University of Chicago(芝加哥大学);Data Science Institute(数据科学研究所);Department of Biochemistry and Molecular Biology(生物化学与分子生物学系);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Argonne National Laboratory(阿贡国家实验室);Princeton University(普林斯顿大学)