Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering（计算机科学与工程系）；Incheon National University（庆尚国立大学）；Department of Artificial Intelligence（人工智能系）；Inha University（Inha大学）；Center for Brain-Machine Interface（脑机接口中心）
Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models
Technical University of Munich（慕尼黑工业大学）；Hong Kong Baptist University（香港浸会大学）；Aalto University（阿尔托大学）；University of Oulu（奥卢大学）；University of Bonn（波恩大学）；EPFL（洛桑联邦理工学院）
Multimodal foundation models are better simulators of the human brain
Renmin University of China（中国人民大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
多模态学习，特别是大规模的多模态预训练，在过去几年中发展迅速，并导致了人工智能（AI）的最大进步。尽管它的有效性，了解多模式预训练模型的潜在机制仍然是一个巨大的挑战。揭示这些模型的可解释性可能会使人工智能领域的新学习范式取得突破。为此，鉴于人类大脑的多模态性质，我们建议借助非侵入性脑成像技术（如功能磁共振成像（fMRI））来探索多模态学习模型的可解释性。具体地说，我们首先提出了一个新设计的多模态基础模型预训练的15万图像-文本对，它已显示出强大的多模态理解和泛化能力，在各种认知下游任务。此外，从神经编码的角度来看（基于我们的基础模型），我们发现多模态训练的视觉和语言编码器与单峰编码器相比更像大脑。特别是，我们确定了一些大脑区域，其中多模态训练的编码器表现出更好的神经编码性能。这与现有探索大脑多感觉整合的研究结果一致。因此，我们认为，多模态基础模型是更适合的工具，神经科学家研究的多模态信号处理机制在人脑中。我们的研究结果还证明了多模态基础模型作为理想的计算模拟器的潜力，以促进AI for Brain和Brain-for-AI研究。
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions
School of Biomedical Engineering, Tsinghua University（清华大学生物医学工程学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua University（清华大学临床医学院北京清华长庚医院肝胆胰中心）；School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；College of Electronic Information Engineering, Tongji University（同济大学电子与信息工程学院）
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
InertialAI；Department of Electrical and Computer Engineering, Queen’s University（皇后大学电气与计算机工程系）；Department of Mechanical and Materials Engineering, Queen’s University（皇后大学机械与材料工程系）
MultiPUFFIN: A Multimodal Domain-Constrained Foundation Model for Molecular Property Prediction of Small Molecules
Department of Chemical Engineering, Norwegian University of Science and Technology (NTNU)（挪威科学与技术大学化学工程系）；Faculty of Industrial Engineering, KU Leuven（鲁文大学工业工程学院）；University of Surrey（萨里大学）
CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer
School of Computer Science and Technology, College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部计算机科学与技术学院）；Data Science Program, Columbian College of Arts and Sciences, The George Washington University（乔治华盛顿大学文理学院哥伦比亚艺术与科学学院数据科学项目）
PolyFusionAgent: A Multimodal Foundation Model and Autonomous AI Assistant for Polymer Property Prediction and Inverse Design
Department of Applied Computer Science, The University of Winnipeg（应用计算机科学系，温尼伯大学）；Department of Mechanical Engineering, University of Manitoba（机械工程系，曼尼托巴大学）
Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Renmin University of China（中国人民大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The Chinese University of Hong Kong（香港中文大学）
A Multimodal Foundation Model to Enhance Generalizability and Data Efficiency for Pan-cancer Prognosis Prediction
Department of Computer Science and Engineering（计算机科学与工程系）；Hong Kong University of Science and Technology（香港科学与技术大学）；Department of Pathology（病理学系）；School of Medicine（医学院）；Zhejiang University（浙江大学）；Nanfang Hospital and School of Basic Medical Sciences（南方医科大学基础医学系）；Southern Medical University（南方医学院）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理重点实验室）；Jinfeng Laboratory（金凤实验室）；Department of Radiology（放射科）；Division of Life Science（生命科学系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）；State Key Laboratory of Nervous System Disorders（神经系统疾病国家重点实验室）
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；The Pennsylvania State University（宾夕法尼亚州立大学）；Beijing Jiaotong University（北京交通大学）；University of Liverpool（利物浦大学）
EyeFound: A Multimodal Generalist Foundation Model for Ophthalmic Imaging
The Hong Kong Polytechnic University（香港理工大学）；Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University（香港理工大学夏普视觉研究中心）；Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑联邦理工学院）；Clemson University（克莱姆森大学）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Singapore Eye Research Institute（新加坡眼科研究所）；State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University（中山大学中山眼科中心眼科学国家重点实验室）；Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science（广东省眼科与视觉科学重点实验室）；Guangdong Provincial Clinical Research Center for Ocular Diseases（广东省眼部疾病临床医学研究中心）；Centre for Eye and Vision Research (CEVR)（眼科与视觉研究中心）
Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery
Guanghua School of Management, Peking University（北京大学光华管理学院）；The Bartlett School of Sustainable Construction, University College London（伦敦大学学院巴特lett可持续建筑学院）；School of Architecture, Tsinghua University（清华大学建筑学院）；School of Computing, The University of Georgia（佐治亚大学计算学院）；Department of Geography, University College London（伦敦大学学院地理系）；Department of Geography and the Environment, University of Texas at Austin（德克萨斯大学奥斯汀分校地理与环境系）
Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics（西南财经大学计算机与人工智能学院）；Beijing Humanoid Robot Innovation Center (X-Humanoid)（北京人形机器人创新中心（X-Humanoid））；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Electronic Science and Technology of China（电子科技大学）
Multimodal Foundation Models For Echocardiogram Interpretation
Cedars-Sinai Medical Center（西达赛奈医疗中心）；University of California Los Angeles（加州大学洛杉矶分校）；University of California, San Francisco（加州大学旧金山分校）；San Francisco Veterans Affairs Medical Center（旧金山退伍军人事务医疗中心）
A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China（深圳大学计算机科学与软件工程学院）；Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology, Shenzhen, 518055, China（视觉对象检测与识别深圳重点实验室）；Laboratory for Artificial Intelligence in Design, Hong Kong（人工智能设计实验室）；School of Artificial Intelligence, Shenzhen University, Shenzhen, China（深圳大学人工智能学院）
VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence
School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；State Key Laboratory of Intelligent Manufacturing System Technology（智能制造系统技术国家重点实验室）；School of Computer Science and Artificial Intelligence, Zhengzhou University（郑州大学计算机科学与人工智能学院）
VSLLaVA: a pipeline of large multimodal foundation model for industrial vibration signal analysis
State Key Laboratory of Tribology, Department of Mechanical Engineering, Tsinghua University（摩擦学国家重点实验室，清华大学机械工程系）；Department of Statistics and Data Science, Yale University（耶鲁大学统计与数据科学系）
MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Chicago（芝加哥大学）；University of California, Berkeley（加州大学伯克利分校）；Harvard University（哈佛大学）；Massachusetts Institute of Technology（麻省理工学院）；Virginia Tech（弗吉尼亚理工大学）；University of Georgia（佐治亚大学）；Microsoft Corporation（微软公司）；Center for AI Safety（人工智能安全中心）
NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；NExT++ Research Center（NExT++研究中心）；National University of Singapore（新加坡国立大学）
Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Harvard Medical School（哈佛医学院）；Stanford University（斯坦福大学）；Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）
Benchmarking Foundation Models with Multimodal Public Electronic Health Records
Centre for Quantitative Medicine and Duke-NUS AI + Medical Science Initiative, Duke-NUS Medical School（定量医学中心和杜克-国立新加坡大学AI+医学科学计划，杜克-国立新加坡大学医学院）；Graduate School of Engineering, The University of Tokyo（东京大学工程研究生院）；Department of Population Health Sciences, Weill Cornell Medicine（流行病学与公共卫生科学系，韦尔·科恩医学中心）；Department of Surgery, Duke University School of Medicine（外科医学系，杜克大学医学学院）；Centre for Quantitative Medicine, Duke-NUS AI + Medical Science Initiative and Programme in Health Services and Systems Research, Duke-NUS Medical School and NUS Artificial Intelligence Institute, National University of Singapore（定量医学中心和杜克-国立新加坡大学AI+医学科学计划及健康服务与系统研究计划，杜克-国立新加坡大学医学院和新加坡国立大学人工智能研究所）
