arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.03276cs.CV
收藏

超越语言模型:多模态预训练的探索

Beyond Language Modeling: An Exploration of Multimodal Pretraining

FAIR, Meta(FAIR、Meta);New York University(纽约大学)
Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan G… 展开作者

Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie

AI总结:

本文通过多模态预训练探索,揭示了视觉与语言数据的互补性及统一预训练对世界建模的促进作用,并提出MoE架构解决多模态扩展的不对称性问题。

摘要:

视觉世界为推动基础模型超越语言提供了关键轴线。尽管在这个方向上兴趣日益增长,但原生多模态模型的设计空间仍然模糊。我们通过受控的从零开始预训练实验提供了实证清晰度,隔离了决定多模态预训练的因素,而不受语言预训练的干扰。我们采用Transfusion框架,使用下一个标记预测进行语言和扩散进行视觉,以训练文本、视频、图像-文本对以及甚至动作条件化的视频等多样化数据。我们的实验得出四个关键见解:(i)表示自动编码器(RAE)通过在视觉理解和生成方面表现出色,提供了最优的统一视觉表示;(ii)视觉和语言数据是互补的,并为下游能力产生协同效应;(iii)统一的多模态预训练自然导致世界建模,能力从一般训练中涌现;(iv)专家混合(MoE)使多模态扩展高效且有效,同时自然诱导模态专业化。通过IsoFLOP分析,我们计算了两种模态的扩展定律,并揭示了扩展不对称性:视觉比语言更需要数据。我们证明,MoE架构通过提供语言所需的高模型容量,同时适应视觉的数据密集型性质,为真正统一的多模态模型铺平了道路。

英文摘要:

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑