Foundation models or pre-trained models have substantially improved the performance of various language, vision, and vision-language understanding tasks. However, existing foundation models can only perform the best in one type of tasks, namely language, vision, or vision-language. It is still an open question whether it is possible to construct a foundation model performing the best for all the understanding tasks, which we call a general foundation model. In this paper, we propose a new general foundation model, X-FM (the X-Foundation Model). X-FM has one language encoder, one vision encoder, and one fusion encoder, as well as a new training method. The training method includes two new techniques for learning X-FM from text, image, and image-text pair data. One is to stop gradients from the vision-language training when learning the language encoder. The other is to leverage the vision-language training to guide the learning of the vision encoder. Extensive experiments on benchmark datasets show that X-FM can significantly outperform existing general foundation models and perform better than or comparable to existing foundation models specifically for language, vision, or vision-language understanding. Code and pre-trained models are released at https://github.com/zhangxinsong-nlp/XFM.
ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts
Institute of Biopharmaceutical and Health Engineering, Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学生物医药与健康工程研究院，清华大学深圳国际研究生院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Department of Pathology, The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院病理科）；Medical Optical Technology R&D Center, Research Institute of Tsinghua, Pearl River Delta（清华珠三角研究院医学光学技术研发中心）
BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis
Paul C. Lauterbur Research Center for Biomedical Imaging（生物医学成像保罗·劳特伯研究中心）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；Pengcheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Faculty of Applied Sciences（应用科学学院）
Florence: A New Foundation Model for Computer Vision
Microsoft Cloud and AI（微软云与人工智能部门）；Microsoft Research Redmond（微软雷德蒙德研究院）
对我们多样化和开放世界的自动视觉理解要求计算机视觉模型能够很好地泛化，并对特定任务进行最小化的定制，类似于人类视觉。计算机视觉基础模型是在各种大规模数据集上训练的，可以适应广泛的下游任务，对于解决现实世界的计算机视觉应用至关重要。虽然现有的视觉基础模型，如CLIP，ALIGN和Wu Dao 2.0主要集中在将图像和文本表示映射到跨模态共享表示，但我们引入了一个新的计算机视觉基础模型Florence，将表示从粗（场景）扩展到细（对象），从静态（图像）扩展到动态（视频），从RGB扩展到多模态（标题，深度）。通过将通用的视觉语言表示从Web规模的图像-文本数据，我们的佛罗伦萨模型可以很容易地适应各种计算机视觉任务，如分类，检索，目标检测，VQA，图像字幕，视频检索和动作识别。此外，Florence在许多类型的迁移学习中表现出色：完全采样微调，线性探测，Few-Shot迁移和新颖图像和对象的zero-shot迁移。所有这些属性对于我们的视觉基础模型服务于通用视觉任务来说都是至关重要的。Florence在44个代表性基准中的大多数中取得了新的最先进的结果，例如，ImageNet-1 K zero-shot分类，top-1精度为83.74，top-5精度为97.18，COCO微调为62.4 mAP，VQA为80.36，Kinetics-600为87.8。
On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital（麻省总医院Athinoula A. Martinos生物医学成像中心）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）
Foundational Models Defining a New Era in Vision: A Survey and Outlook
MBZ University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；University of Central Florida（中佛罗里达大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation
Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉A.马丁诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；University Campus Bio-Medico of Rome（罗马生物医学大学校园）
SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality
Center for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人中心）；Princeton University（普林斯顿大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Washington（华盛顿大学）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究所）；Chinese Academy of Sciences（中国科学院）；State Key Laboratory of Ophthalmology（眼科学国家重点实验室）；Zhongshan Ophthalmic Center（中山眼科中心）；Sun Yat-sen University（中山大学）；Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science（广东省眼科学与视觉科学重点实验室）；Guangdong Provincial Clinical Research Center for Ocular Diseases（广东省眼科临床研究中心）；Department of Bioengineering（生物工程系）；Department of Radiology（放射科）
