The Design Space of Tri-Modal Masked Diffusion Models
Apple(苹果公司);Google Deepmind(谷歌DeepMind);University of Cambridge(剑桥大学);MIT(麻省理工学院)
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec, Pau Rodriguez Lopez, Lokesh Boominathan, Nikhil Bhendawade, Amitis Shidani, Joris Pelemans, Theo X. Ola… 展开作者
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec, Pau Rodriguez Lopez, Lokesh Boominathan, Nikhil Bhendawade, Amitis Shidani, Joris Pelemans, Theo X. Olausson, Devon Hjelm, Paul Dixon, Joao Monteiro, Pierre Ablin, Vishnu Banna, Arno Blaas, Nick Henderson, Kari Noriy, Dan Busbridge, Josh Susskind, Marco Cuturi, Irina Belousova, Luca Zappella, Russ Webb, Jason Ramapuram
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal generation. Diverging from previous approaches, we introduce the first tri-modal masked diffusion model pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and we provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation (SDE)-based reparameterization that eliminates the need for tuning the optimal batch size as reported in recent work. This reparameterization decouples the physical batch size, often chosen based on compute constraints (GPU saturation, FLOP efficiency, wall-clock time), from the logical batch size, chosen to balance gradient variance during stochastic optimization. Finally, we pretrain a preliminary 3B-parameter tri-modal model on 6.4T tokens, demonstrating the capabilities of a unified design and achieving strong results in text generation, text-to-image tasks, and text-to-speech tasks. Our work represents the largest-scale systematic open study of multimodal discrete diffusion models conducted to date, providing insights into scaling behaviors across multiple modalities.
Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition
MoCA: Multi-modal Cross-masked Autoencoder for Time Series in Digital Health
Herbert Wertheim School of Public Health and Human Longevity Science, University of California, San Diego, San Diego, CA, USA(赫伯特·韦瑟姆公共卫生与人类长寿科学学院,加州大学圣地亚哥分校);Halıcıoğlu Data Science Institute, University of California San Diego, San Diego, CA, USA(哈利奇奥卢数据科学研究所,加州大学圣地亚哥分校);Department of Computer Science and Engineering, University of California San Diego, San Diego, CA, USA(计算机科学与工程系,加州大学圣地亚哥分校);Division of Public Health Sciences, Fred Hutchinson Cancer Center, Seattle, WA, USA(公共卫生科学部,Fred Hutchinson癌症中心)
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
School of Informatics, Xiamen University(厦门大学信息学院);Center for Smart Health, School of Nursing, The Hong Kong Polytechnic University(香港理工大学护理学院智能健康中心);Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系);Department of Radiology, Zhongshan Hospital (Xiamen), Fudan University(复旦大学中山医院放射科);National Institute for Data Science in Health and Medicine(医学与数据科学国家研究院);Department of Computer Science, School of Informatics, Xiamen University(厦门大学信息学院计算机科学系)
ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding
NAVER CLOVA;NAVER AI LAB(NAVER人工智能实验室)
语言模型预训练在各种下游任务中显示出有希望的结果。在这种情况下,我们引入了一个跨模态的预训练语言模型,称为语音-文本BERT(ST-BERT),以解决端到端口语语言理解(E2 E SLU)任务。以音素后验和子词级文本为输入,ST-BERT通过我们提出的两个预训练任务:跨模态掩蔽语言建模(CM-MLM)和跨模态条件语言建模(CM-CLM)来学习上下文化的跨模态对齐。三个基准测试的实验结果表明,我们的方法是有效的各种SLU数据集,并显示出令人惊讶的边际性能下降,即使当1%的训练数据可用。此外,我们的方法通过使用特定领域的语音-文本对数据进行领域自适应预训练,进一步提高了SLU性能。