Current multimodal and multitask foundation models like 4M or UnifiedIO show promising results, but in practice their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually rather small) number of modalities and tasks they are trained on. In this paper, we expand upon the capabilities of them by training a single model on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example image metadata or color palettes. A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text. Through this, we expand on the out-of-the-box capabilities of multimodal models and specifically show the possibility of training one model to solve at least 3x more tasks/modalities than existing ones and doing so without a loss in performance. This enables more fine-grained and controllable multimodal generation capabilities and allows us to study the distillation of models trained on diverse data and objectives into a unified model. We successfully scale the training to a three billion parameter model using tens of modalities and different datasets. The resulting models and training code are open sourced at 4m.epfl.ch.
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））
VLC Fusion: Vision-Language Conditioned Sensor Fusion for Robust Object Detection
Arizona State University（亚利桑那州立大学）；Department of Computer Science and Engineering, Universidad Nacional del Sur and Institute for Computer Science and Engineering（计算机科学与工程系，国家南方大学和计算机科学与工程研究所）；U.S. Department of Defense（美国国防部）；United States Military Academy（美国军事学院）；Syracuse University（雪城大学）
Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers
Heidelberg University Hospital（海德堡大学医院）；Informatics for Life Institute（生命信息学研究所）；German Centre for Cardiovascular Research（德国心血管研究中心）；University Medical Center of the Johannes Gutenberg University Mainz（美因茨约翰内斯·古滕贝格大学医学中心）；University Medical Center Mainz（美因茨大学医学中心）；Institute of Molecular Biology (IMB)（分子生物学研究所）
Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities
School of Computer Science and Engineering, Southeast University, China（计算机科学与工程学院，东南大学，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China（新一代人工智能技术及其交叉应用重点实验室，教育部，中国）；Nanjing University of Science and Technology, China（南京理工大学，中国）
Northwestern Polytechnical University（西北工业大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；The University of Hong Kong（香港大学）
Unifying Vision, Text, and Layout for Universal Document Processing
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Microsoft Azure Cognitive Services Research（微软Azure认知服务研究院）；Microsoft Azure Visual Document Intelligence（微软Azure视觉文档智能部门）
Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?
Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学）；Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Oulu（奥卢大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；Brno University of Technology（布尔诺理工大学）
DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models
Harvard University（哈佛大学）；City University of Hong Kong（香港城市大学）；Amazon（亚马逊）；Fudan University（复旦大学）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能关键实验室，上海交通大学）
MBQ: Modality-Balanced Quantization for Large Vision-Language Models
Tsinghua University（清华大学）；Infinigence-AI（云知声人工智能）；Beihang University（北京航空航天大学）；Qingdao Research Institute, Beihang University（北京航空航天大学青岛研究院）；University of Hong Kong（香港大学）；Chinatower（中国铁塔）；Shanghai Jiao Tong University（上海交通大学）
PainFormer: a Vision Foundation Model for Automatic Pain Assessment
Hellenic Mediterranean University, Department of Electrical and Computer Engineering（希腊地中海大学电子与计算机工程系）；Institute of Computer Science, Foundation for Research & Technology-Hellas（希腊研究所计算机科学研究所）；University of Canberra, Faculty of Science and Technology（堪培拉大学科学与技术学院）
