We propose a scaling law hypothesis for multimodal models processing text, audio, images, and video within a shared token and embedding space. Our framework predicts model performance based on modality-specific compression and tokenization efficiency, extending established scaling laws from text-based decoder models to mixed-modality systems. We explore whether leveraging more training data in multiple modalities can reduce the size of the multimodal model, enabling efficient deployment on resource-constrained devices.
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
School of Software, Tsinghua University, Beijing, China(清华大学软件学院,北京,中国);TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院,中国);School of New Media and Communication, Tianjin University, China(天津大学新媒体与传播学院,中国)
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
University of Washington(华盛顿大学);Salesforce Research(Salesforce研究院);University of Texas at Austin(德克萨斯大学奥斯汀分校);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校)
Test-Time Scaling Strategies for Generative Retrieval in Multimodal Conversational Recommendations
Research Center for Information Technology Innovation, Academia Sinica(资讯科技创新研究所以);NVIDIA(NVIDIA公司);Department of Computer Science, National Chengchi University(国立政治大学计算机科学系)
Hypothesis-and-Refinement Learning of Organic Structures from Multimodal Spectroscopic Data
State Key Laboratory of Advanced Waterproof Materials, School of Materials Science and Engineering, Peking University(先进防水材料国家重点实验室,材料科学与工程学院,北京大学);School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School(电子与计算机工程学院,北京大学深圳研究生院);Peng Cheng Laboratory(鹏城实验室);IBS Center for Algorithmic and Robotized Synthesis (CARS), UNIST(算法与机器人化合成中心(CARS),UNIST);Department of Chemistry, UNIST(化学系,UNIST);School of Advanced Materials, Peking University Shenzhen Graduate School(先进材料学院,北京大学深圳研究生院)
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Massachusetts Institute of Technology(麻省理工学院);South China University of Technology(华南理工大学);Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院)
MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
Yale University(耶鲁大学);Korea University(韩国大学);Washington University in St. Louis(圣路易斯华盛顿大学);The University of Queensland(昆士兰大学);The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);University of Washington(华盛顿大学);Microsoft Research(微软研究院)