Training large language models (LLMs) and multimodal LLMs necessitates significant computing resources, and existing publicly available LLMs are typically pre-trained on diverse, privately curated datasets spanning various tasks. For instance, LLaMA, Vicuna, and LLaVA are three LLM variants trained with LLaMA base models using very different training recipes, tasks, and data modalities. The training cost and complexity for such LLM variants grow rapidly. In this study, we propose to use a soup strategy to assemble these LLM variants into a single well-generalized multimodal LLM (SoupLM) in a cost-efficient manner. Assembling these LLM variants efficiently brings knowledge and specialities trained from different domains and data modalities into an integrated one (e.g., chatbot speciality from user-shared conversations for Vicuna, and visual capacity from vision-language data for LLaVA), therefore, to avoid computing costs of repetitive training on several different domains. We propose series of soup strategies to systematically benchmark performance gains across various configurations, and probe the soup behavior across base models in the interpolation space.
Unified Molecule Pre-training with Flexible 2D and 3D Modalities: Single and Paired Modality Integration
Computational Bioscience Research Center, King Abdullah University of Science and Technology（国王阿卜杜勒·阿齐兹科技大学计算生物科学研究中心）；Institute for Infocomm Research, A*STAR（信息与通信研究机构，A*STAR）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算机与信息系统学院）
Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs
Fondazione Bruno Kessler（布鲁诺·凯瑟尔基金会）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；Soongsil University（顺天大学）；Charles University（查尔斯大学）；KIT（卡尔斯鲁厄理工学院）
Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory（中文信息处理实验室）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
Robust Multi-Omics Integration from Incomplete Modalities Significantly Improves Prediction of Alzheimer's Disease
LG AI Research, Seoul, Republic of Korea（LG AI研究院，韩国首尔）；Department of Chemistry, Seoul National University, Seoul, Republic of Korea（首尔国立大学化学系）；The Jackson Laboratory, Bar Harbor, Maine, USA（杰克逊实验室，缅因州巴尔 Harbor）
StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation
University of Science and Technology of China（科学技术大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究所（TeleAI），中国电信）；Northwestern Polytechnical University（西北工业大学）
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of Science and Technology of China（中国科学技术大学）；Huazhong University of Science and Technology（华中科技大学）；Microsoft Research（微软研究院）
Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration
Hatyaiwittayalai School（哈雅维泰莱学校）；School of Information, Computer, and Communication Technology（信息、计算机与通信技术学院）；Sirindhorn International Institute of Technology（Sirindhorn国际技术学院）；Thammasat University（泰国朱拉大学）；Pathum Thani, Thailand（帕亨他尼，泰国）
Hierarchical Federated Foundation Models over Wireless Networks for Multi-Modal Multi-Task Intelligence: Integration of Edge Learning with D2D/P2P-Enabled Fog Learning Architectures
Multi-modal Multi-kernel Graph Learning for Autism Prediction and Biomarker Discovery
Central South University（中南大学）；Xinjiang University（新疆大学）；Shenzhen University of Advanced Technology（深圳先进技术研究院？不，正确翻译：深圳理工大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Dundee（邓迪大学）；Griffith University（格里菲斯大学）；Shenzhen Health Development Research and Data Management Center（深圳卫生发展研究与数据管理中心）
A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）
MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention
The University of Sydney（悉尼大学）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（集成空天地海大数据应用技术国家工程实验室）；School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机科学与工程学院）；University of Maryland（马里兰大学）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波学院）
Benchmarking Preprocessing and Integration Methods in Single-Cell Genomics
University of Technology Sydney, Australia（悉尼科技大学）；University of Sydney, Australia（悉尼大学）；Broad Institute, United States（Broad研究所）；University of New South Wales, Australia（新南威尔士大学）；Shaoyang University, China（韶阳大学）
Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization
Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；Department of Health Technology & Informatics, The Hong Kong Polytechnic University（香港理工大学健康科技与信息学系）；Department of Statistics and Actuarial Science, The University of Hong Kong（香港大学统计与精算科学系）；Department of Clinical Neurosciences and Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学临床神经科学系和应用数学与理论物理系；邓迪大学科学与工程学院和医学院）；School of Science and Engineering and School of Medicine, University of Dundee, UK
