持续多模态预训练实践者指南
A Practitioner's Guide to Continual Multimodal Pretraining
AI总结：
本文提出 FoMo-in-Flux 基准，基于 63 个数据集系统评估数据、方法、调度及规模因素，为真实部署中的持续多模态预训练提供实践指南。
摘要：
多模态基础模型服务于视觉与语言交叉领域的众多应用。然而，尽管它们已在海量数据上完成预训练，仍会随时间推移而过时。为保持模型更新，关于持续预训练的研究主要探索两类场景：（1）在大规模新数据上进行不频繁、不加区分的更新，或（2）频繁的样本级更新。但实际模型部署往往处于这两种极端情形之间，因为真实世界应用通常要求适应特定子领域、任务或概念，而这些需求贯穿模型整个不断变化的生命周期。在本工作中，我们通过一个研究测试平台补充当前关于持续预训练的视角，并为此类场景下有效的持续模型更新提供全面指导。我们首先介绍 FoMo-in-Flux，这是一个具有现实计算约束和实际部署要求的持续多模态预训练基准，基于 63 个具有多样化视觉和语义覆盖的数据集构建。利用 FoMo-in-Flux，我们从多个视角探索实际持续预训练的复杂图景：（1）以数据为中心，考察模拟真实部署情形的数据混合和流顺序；（2）以方法为中心，考察范围从简单微调和传统持续学习策略到参数高效更新及模型合并；（3）元学习率调度和机制设计选择；以及（4）模型与计算规模扩展的影响。综合而言，我们的洞见为面向真实部署的持续多模态预训练提供了一份实践者指南。我们的基准和代码位于：https://github.com/ExplainableML/fomo_in_flux。
英文摘要：
Multimodal foundation models serve numerous applications at the intersection of vision and language. Still, despite being pretrained on extensive data, they become outdated over time. To keep models updated, research into continual pretraining mainly explores scenarios with either (1) infrequent, indiscriminate updates on large-scale new data, or (2) frequent, sample-level updates. However, practical model deployment often operates in the gap between these two limit cases, as real-world applications often demand adaptation to specific subdomains, tasks or concepts -- spread over the entire, varying life cycle of a model. In this work, we complement current perspectives on continual pretraining through a research test bed as well as provide comprehensive guidance for effective continual model updates in such scenarios. We first introduce FoMo-in-Flux, a continual multimodal pretraining benchmark with realistic compute constraints and practical deployment requirements, constructed over 63 datasets with diverse visual and semantic coverage. Using FoMo-in-Flux, we explore the complex landscape of practical continual pretraining through multiple perspectives: (1) A data-centric investigation of data mixtures and stream orderings that emulate real-world deployment situations, (2) a method-centric investigation ranging from simple fine-tuning and traditional continual learning strategies to parameter-efficient updates and model merging, (3) meta learning rate schedules and mechanistic design choices, and (4) the influence of model and compute scaling. Together, our insights provide a practitioner's guide to continual multimodal pretraining for real-world deployment. Our benchmark and code is here: https://github.com/ExplainableML/fomo_in_flux.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
