Beyond Language Modeling: An Exploration of Multimodal Pretraining
FAIR, Meta(FAIR、Meta);New York University(纽约大学)
Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan G… 展开作者
Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
Emerging Properties in Unified Multimodal Pretraining
ByteDance Seed(字节跳动种子);Shenzhen Institutes of Advanced Technology(深圳先进技术研究院);Monash University(墨尔本大学);Hong Kong University of Science and Technology(香港科学与技术大学);UC Santa Cruz(加州大学圣克ruz分校)
Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
University of Lincoln, Lincoln Centre for Autonomous Systems(林肯大学,林肯自主系统中心);University of Cambridge, Institute for Manufacturing, Department of Engineering(剑桥大学,制造研究所,工程系)
Multilingual Multimodal Learning with Machine Translated Text
Wuhan University of Science and Technology(武汉科技大学);University Politehnica of Bucharest(布加勒斯特理工大学);University of Copenhagen(哥本哈根大学);Pioneer Centre for AI(人工智能先锋中心)
Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA
Cohere for AI Community(Cohere for AI社区);Cisco Meraki;Indiana University Bloomington(印第安纳大学布卢明顿分校);Bangladesh University of Engineering and Technology(孟加拉工程与技术大学)
InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
Alibaba Group(阿里巴巴集团);Peking University(北京大学)
用于学习高级多模态表示的多模态预训练是迈向深度学习和人工智能的又一步。在这项工作中,我们提出了一个新的模型,即InterBERT(BERT for Interaction),这是我们的多模态预训练方法M6(MultiModality-to-MultiModality Multitask Mega-transformer)系列的第一个模型。该模型具有很强的建模能力,不同模态的信息流之间的相互作用。单流交互模块能够有效地处理多模态的信息,而上层的双流交互模块保持了各模态的独立性,避免了单模态任务的性能下降。我们使用三个预训练任务对模型进行预训练,包括掩蔽段建模(MSM)、掩蔽区域建模(MRM)和图像-文本匹配(ITM);并在一系列视觉和语言下游任务上对模型进行微调。实验结果表明,InterBERT的性能优于一系列强基线,包括最新的多模态预训练方法,分析表明MSM和MRM是有效的预训练方法,我们的方法在单模态任务中可以达到与BERT相当的性能。此外,我们提出了一个大规模的数据集,用于中文多模态预训练,我们开发了中文InterBERT,这是第一个中文多模态预训练模型。我们在我们提出的来自中国最大的电子商务平台手机淘宝的310万图像-文本对数据集上预训练了中文InterBERT。我们微调的模型,基于文本的图像检索,最近,我们部署了基于主题的推荐模型在线。
A Practitioner's Guide to Continual Multimodal Pretraining
Tübingen AI Center(蒂宾根人工智能中心);University of Tübingen(蒂宾根大学);Helmholtz Munich(亥姆霍兹慕尼黑中心);University of Cambridge(剑桥大学);LAION;Jülich Supercomputing Center (JSC)(于利希超级计算中心);Research Center Jülich (FZJ)(于利希研究中心);Helmholtz Association(亥姆霍兹联合会);Google DeepMind(谷歌DeepMind);Munich Center for ML(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学)
Beijing University of Posts and Telecommunications(北京邮电大学);Rutgers University(罗格斯大学);Shanghai AI Laboratory(上海人工智能实验室);Hasso Plattner Institute, University of Potsdam(波茨坦大学哈索普拉特纳研究所);YITU Technology(依图科技)
Multimodal Contrastive Pretraining of CBCT and IOS for Enhanced Tooth Segmentation
Department of Computer Science and Engineering(计算机科学与工程系);The Hong Kong University of Science and Technology(香港科学与技术大学);Division of Paediatric Dentistry and Orthodontics, Faculty of Dentistry(牙科学院儿童牙科与正畸科);The University of Hong Kong(香港大学);Delun Dental Hospital(德尔伦牙科医院);Department of Chemical and Biological Engineering(化学与生物工程系);Division of Life Science(生命科学系);HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute(香港科技大学深圳-香港协同创新研究院);State Key Laboratory of Nervous System Disorders(神经系统紊乱国家重点实验室)
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
UMIT TIROL – Private University for Health Sciences and Health Technology(UMIT TIROL – 健康科学与健康技术私立大学);VASCage – Centre on Clinical Stroke Research(VASCage – 临床卒中研究中心);Medical University of Innsbruck(因斯布鲁克医科大学)
Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology(北京智能信息科技重点实验室,计算机科学与技术学院,北京理工大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);State Key Laboratory of General Artificial Intelligence, Peking University(通用人工智能国家重点实验室,北京大学);Harbin Institute of Technology(哈尔滨工业大学);Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University(广东机器感知与智能计算实验室,深圳MSU-BIT大学);Department of Automation, Tsinghua University(自动化系,清华大学)
SCAFusion: A Multimodal 3D Detection Framework for Small Object Detection in Lunar Surface Exploration
Department of Automation, Key Laboratory of System Control and Information Processing of Ministry of Education, Key Laboratory of Marine Intelligent Equipment and System of Ministry of Education, Shanghai Engineering Research Center of Intelligent Control and Management, Shanghai Jiao Tong University(自动化系、教育部系统控制与信息处理重点实验室、教育部海洋智能装备与系统重点实验室、上海智能控制与管理工程研究中心、上海交通大学)
Explore the Limits of Omni-modal Pretraining at Scale
The Chinese University of Hong Kong(香港中文大学);University of Chinese Academy of Sciences(中国科学院大学);Chinese Academy of Science(中国科学院);Shanghai AI Laboratory(上海人工智能实验室)