In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. We show here how scaling law derivation can also be used for model and dataset comparison, allowing to decide which procedure is to be preferred for pre-training. For the first time, full scaling laws based on dense measurements across a wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss. Ensuring sufficient prediction accuracy for held out points, we use derived scaling laws to compare both models, obtaining evidence for MaMMUT's stronger improvement with scale and better sample efficiency than standard CLIP. To strengthen validity of the comparison, we show scaling laws for various downstream tasks, classification, retrieval, and segmentation, and for different open datasets, DataComp, DFN and Re-LAION, observing consistently the same trends. We show that comparison can also be performed when deriving scaling laws with a constant learning rate schedule, reducing compute cost. Accurate derivation of scaling laws provides thus means to perform model and dataset comparison across scale spans, avoiding misleading conclusions based on measurements from single reference scales only, paving the road for systematic comparison and improvement of open foundation models and datasets for their creation. We release all the pre-trained models with their intermediate checkpoints, including openMaMMUT-L/14, which achieves $80.3\%$ zero-shot ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for reproducing experiments in the paper and raw experiments data can be found at https://github.com/LAION-AI/scaling-laws-for-comparison.
BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis
Paul C. Lauterbur Research Center for Biomedical Imaging(生物医学成像保罗·劳特伯研究中心);Shenzhen Institutes of Advanced Technology(深圳先进技术研究院);Chinese Academy of Sciences(中国科学院);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Faculty of Applied Sciences(应用科学学院)
UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound
University of California, Los Angeles(加州大学洛杉矶分校);Université Laval(拉瓦尔大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);UCLA David Geffen School of Medicine(加州大学洛杉矶分校大卫·格芬医学院)
SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality
Center for Artificial Intelligence and Robotics, HKISI, CAS(中国科学院香港创新研究院人工智能与机器人中心);Princeton University(普林斯顿大学);The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室)
OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation
Athinoula A. Martinos Center for Biomedical Imaging(阿提诺拉A.马丁诺斯生物医学成像中心);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);University Campus Bio-Medico of Rome(罗马生物医学大学校园)
VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Washington(华盛顿大学);Shenzhen Institutes of Advanced Technology(深圳先进技术研究所);Chinese Academy of Sciences(中国科学院);State Key Laboratory of Ophthalmology(眼科学国家重点实验室);Zhongshan Ophthalmic Center(中山眼科中心);Sun Yat-sen University(中山大学);Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science(广东省眼科学与视觉科学重点实验室);Guangdong Provincial Clinical Research Center for Ocular Diseases(广东省眼科临床研究中心);Department of Bioengineering(生物工程系);Department of Radiology(放射科)
SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality
Center for Artificial Intelligence and Robotics, HKISI, CAS(中国科学院香港中文大学人工智能机器人中心);Princeton University(普林斯顿大学);The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室)
SatelliteCalculator: A Multi-Task Vision Foundation Model for Quantitative Remote Sensing Inversion
Faculty of Computer Science and Information Technology, Universiti Malaya(马来亚大学计算机科学与信息技术学院);Faculty of Information Engineering and Automation, Kunming University of Science and Technology(昆明理工大学信息工程与自动化学院)
Cross-Domain Foundation Model Adaptation: Pioneering Computer Vision Models for Geophysical Data Analysis
School of Earth and Space Sciences, University of Science and Technology of China(中国科学技术大学地球和空间科学学院);Microsoft Applied Sciences Group(微软应用科学组);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)
ZeroSlide: Is Zero-Shot Classification Adequate for Lifelong Learning in Whole-Slide Image Analysis in the Era of Pathology Vision-Language Foundation Models?
Nara Institute of Science and Technology(奈良科学技术研究所)
Florence: A New Foundation Model for Computer Vision
Microsoft Cloud and AI(微软云与人工智能部门);Microsoft Research Redmond(微软雷德蒙德研究院)
对我们多样化和开放世界的自动视觉理解要求计算机视觉模型能够很好地泛化,并对特定任务进行最小化的定制,类似于人类视觉。计算机视觉基础模型是在各种大规模数据集上训练的,可以适应广泛的下游任务,对于解决现实世界的计算机视觉应用至关重要。虽然现有的视觉基础模型,如CLIP,ALIGN和Wu Dao 2.0主要集中在将图像和文本表示映射到跨模态共享表示,但我们引入了一个新的计算机视觉基础模型Florence,将表示从粗(场景)扩展到细(对象),从静态(图像)扩展到动态(视频),从RGB扩展到多模态(标题,深度)。通过将通用的视觉语言表示从Web规模的图像-文本数据,我们的佛罗伦萨模型可以很容易地适应各种计算机视觉任务,如分类,检索,目标检测,VQA,图像字幕,视频检索和动作识别。此外,Florence在许多类型的迁移学习中表现出色:完全采样微调,线性探测,Few-Shot迁移和新颖图像和对象的zero-shot迁移。所有这些属性对于我们的视觉基础模型服务于通用视觉任务来说都是至关重要的。Florence在44个代表性基准中的大多数中取得了新的最先进的结果,例如,ImageNet-1 K zero-shot分类,top-1精度为83.74,top-5精度为97.18,COCO微调为62.4 mAP,VQA为80.36,Kinetics-600为87.8。
Towards Neural Scaling Laws for Time Series Foundation Models
Griffith University(格里菲斯大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));NVIDIA Research(英伟达研究院);The University of Tokyo(东京大学)
CellVTA: Enhancing Vision Foundation Models for Accurate Cell Segmentation and Classification
ShanghaiTech University(上海科技大学);Shanghai Ocean University(上海海洋大学);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心)
Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging
Massachusetts Institute of Technology(麻省理工学院);University of California, San Diego(加州大学圣地亚哥分校);University of Washington(华盛顿大学);University of Washington School of Medicine(华盛顿大学医学院)
VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbones
Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);State Street Technology (Zhejiang) Ltd.(State Street Technology(浙江)有限公司);Salesforce Research Asia(Salesforce亚洲研究)
Pedestrian Intention Prediction via Vision-Language Foundation Models
Institute for Transport Studies, Faculty of Environment, Computer Vision and Machine Learning Group, University of Leeds(运输研究学院、环境学院、计算机视觉与机器学习小组、莱斯特大学);AI Centre, Department of Computer Science, University College London(人工智能中心、计算机科学系、伦敦大学学院)
Cross-modal learning for SAR target recognition using optical vision foundation models
School of Engineering, University of Edinburgh(爱丁堡大学工程学院);Leonardo UK(英国莱昂纳多公司);School of Engineering and Physical Sciences, Heriot-Watt University(赫瑞瓦特大学工程与物理科学学院)