专用基础模型难以击败有监督基线
Specialized Foundation Models Struggle to Beat Supervised Baselines
AI总结:
该研究针对基因组学、卫星成像、时间序列三个专用领域,对比近期基础模型与仅用目标任务数据训练的简单有监督模型,发现后者性能可匹配甚至超越前者,指出专用领域大规模预训练优势尚未显现,并推出两款用于基线对比的自动化开源工作流。
摘要:
继在视觉和文本领域取得成功后,“基础模型”(FM)范式——在海量数据上预训练大模型,再针对目标任务进行微调——已迅速扩展到科学、工程、医疗保健等领域。这种范式是否实现了原始基础模型的成效,即在其所属领域取代传统有监督学习?为回答这一问题,我们研究了基因组学、卫星成像和时间序列三种模态,选取多个近期提出的基础模型,将其与标准有监督学习工作流进行对比:后者的模型开发、超参数调优和训练均仅使用目标任务的数据。在这三个专用领域中,我们发现始终可以训练出简单的有监督模型——复杂程度不超过经轻度修改的宽ResNet或UNet——其性能能够匹配甚至超越最新的基础模型。本研究表明,大规模预训练的优势在许多专业领域尚未显现,同时强调了将新型基础模型与经过充分调优的强基线进行对比的必要性,并为此推出了两种全新、易用、开源且自动化的工作流。
英文摘要:
Following its success for vision and text, the "foundation model" (FM) paradigm -- pretraining large models on massive data, then fine-tuning on target tasks -- has rapidly expanded to domains in the sciences, engineering, healthcare, and beyond. Has this achieved what the original FMs accomplished, i.e. the supplanting of traditional supervised learning in their domains? To answer we look at three modalities -- genomics, satellite imaging, and time series -- with multiple recent FMs and compare them to a standard supervised learning workflow: model development, hyperparameter tuning, and training, all using only data from the target task. Across these three specialized domains, we find that it is consistently possible to train simple supervised models -- no more complicated than a lightly modified wide ResNet or UNet -- that match or even outperform the latest foundation models. Our work demonstrates that the benefits of large-scale pretraining have yet to be realized in many specialized areas, reinforces the need to compare new FMs to strong, well-tuned baselines, and introduces two new, easy-to-use, open-source, and automated workflows for doing so.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。