探索大规模全模态预训练的极限
Explore the Limits of Omni-modal Pretraining at Scale
AI总结：
本文提出可扩展预训练范式 MiCo，通过同步扩展模态、数据与模型参数构建全模态智能，并在多类基准上取得37项新的最先进纪录。
摘要：
我们提出构建全模态智能，使其能够理解任意模态并学习通用表示。具体而言，我们提出一种可扩展的预训练范式，名为 Multimodal Context（MiCo），它能够在预训练过程中同步扩展模态数量、数据规模以及模型参数。借助 MiCo，预训练模型在多模态学习中展现出显著的涌现能力，并在以下任务上接受评估：i）覆盖10种不同模态的单模态感知基准，ii）25项涉及检索、问答和描述生成的跨模态理解任务，以及 iii）18个多模态大语言模型基准。我们的模型在最先进性能上创造了37项新纪录。我们希望本研究能够推动全模态智能的发展。代码和模型位于 https://github.com/invictus717/MiCo。
英文摘要：
We propose to build omni-modal intelligence, which is capable of understanding any modality and learning universal representations. In specific, we propose a scalable pretraining paradigm, named Multimodal Context (MiCo), which can scale up the numbers of modalities and amount of data, together with the model parameters, in the pretraining process. With MiCo, the pretrained models show significant emergent abilities in multimodal learning, which are evaluated on the following tasks: i) single-modality perception benchmarks of 10 different modalities, ii) 25 cross-modality understanding tasks of retrieval, question-answering, captioning, and iii) 18 multimodal large language model benchmarks. Our models establish 37 new records for state-of-the-art performance. We hope that our research could contribute to the development of omni-modal intelligence. Code and Models are at https://github.com/invictus717/MiCo
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
