arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21472cs.LG
收藏

三模态掩码扩散模型的设计空间

The Design Space of Tri-Modal Masked Diffusion Models

Apple(苹果公司);Google Deepmind(谷歌DeepMind);University of Cambridge(剑桥大学);MIT(麻省理工学院)
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec, Pau Rodriguez Lopez, Lokesh Boominathan, Nikhil Bhendawade, Amitis Shidani, Joris Pelemans, Theo X. Ola… 展开作者

Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec, Pau Rodriguez Lopez, Lokesh Boominathan, Nikhil Bhendawade, Amitis Shidani, Joris Pelemans, Theo X. Olausson, Devon Hjelm, Paul Dixon, Joao Monteiro, Pierre Ablin, Vishnu Banna, Arno Blaas, Nick Henderson, Kari Noriy, Dan Busbridge, Josh Susskind, Marco Cuturi, Irina Belousova, Luca Zappella, Russ Webb, Jason Ramapuram

AI总结:

本文提出了一种从头开始预训练的三模态掩码扩散模型,通过分析多模态扩展定律和批大小效应,实现了优化的推理采样,并在文本生成、图像生成和语音生成任务中取得了显著成果。

摘要:

离散扩散模型已经 emerged 作为自回归语言模型的强大替代品,最近的研究工作通过初始化和微调一个基础单模态模型来实现双模态生成。与之前的方法不同,我们首次引入了从文本、图像-文本和音频-文本数据从头开始预训练的三模态掩码扩散模型。我们系统地分析了多模态扩展定律、模态混合比例、噪声调度和批大小效应,并提供了优化的推理采样默认设置。我们的批大小分析产生了一种新的随机微分方程(SDE)基于重参数化,消除了最近工作中报告的需要调整最优批大小的必要性。这种重参数化将物理批大小(通常基于计算限制(GPU饱和、FLOP效率、墙钟时间)选择)与逻辑批大小(用于在随机优化中平衡梯度方差)分离开。最后,我们在6.4T tokens上预训练了一个初步的3B参数三模态模型,展示了统一设计的能力,并在文本生成、文本到图像任务和文本到语音任务中取得了强劲的结果。我们的工作代表了迄今为止对多模态离散扩散模型进行的最大规模系统开放研究,提供了跨多种模态的扩展行为的见解。

英文摘要:

Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal generation. Diverging from previous approaches, we introduce the first tri-modal masked diffusion model pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and we provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation (SDE)-based reparameterization that eliminates the need for tuning the optimal batch size as reported in recent work. This reparameterization decouples the physical batch size, often chosen based on compute constraints (GPU saturation, FLOP efficiency, wall-clock time), from the logical batch size, chosen to balance gradient variance during stochastic optimization. Finally, we pretrain a preliminary 3B-parameter tri-modal model on 6.4T tokens, demonstrating the capabilities of a unified design and achieving strong results in text generation, text-to-image tasks, and text-to-speech tasks. Our work represents the largest-scale systematic open study of multimodal discrete diffusion models conducted to date, providing insights into scaling behaviors across multiple modalities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑