We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for future research in terms of data scaling, training setups and model selections. Our model weights and codes will be made public.
Quantized-Tinyllava: a new multimodal foundation model enables efficient split learning
Department of Statistics University of Michigan(统计学系密歇根大学);Department of Computational Medicine & Bioinformatics University of Michigan(计算医学与生物信息学系密歇根大学);Department of Biostatistics University of Michigan(生物统计学系密歇根大学);Department of Computer Science University of California, Los Angeles(计算机科学系加州大学洛杉矶分校)
Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis
School of Computer Science, South China Normal University(计算机科学学院,华南师范大学);School of Artificial Intelligence, South China Normal University(人工智能学院,华南师范大学);School of Electronics and Information Technology, Sun Yat-sen University(电子与信息工程学院,中山大学)
A Novel Multimodal Approach for Studying the Dynamics of Curiosity in Small Group Learning
ETH Zürich(苏黎世联邦理工学院);University of Rochester(罗切斯特大学);Carnegie Mellon University(卡内基梅隆大学);Paris Artificial Intelligence Research Institute, Inria(法国国家信息与自动化研究所巴黎人工智能研究院)
RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants
Singapore University of Technology and Design(新加坡科技设计大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Nanyang Technological University(南洋理工大学);Chongqing University(重庆大学)
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
Research Unit of Computer Systems and Bioinformatics, Department of Engineering(计算机系统与生物信息学研究单位,工程系);Research Unit of Radiation Oncology, Department of Medicine and Surgery(放射肿瘤学研究单位,医学与外科系);Research Unit of Anatomical Pathology, Department of Medicine and Surgery(病理学研究单位,医学与外科系)
MultiPUFFIN: A Multimodal Domain-Constrained Foundation Model for Molecular Property Prediction of Small Molecules
Department of Chemical Engineering, Norwegian University of Science and Technology (NTNU)(挪威科学与技术大学化学工程系);Faculty of Industrial Engineering, KU Leuven(鲁文大学工业工程学院);University of Surrey(萨里大学)
Predicting Targeted Therapy Resistance in Non-Small Cell Lung Cancer Using Multimodal Machine Learning
Geisel School of Medicine at Dartmouth(达特茅斯盖泽尔医学院);Ochsner Health System(奥克斯纳医疗系统);Leiden University Medical Center(莱顿大学医学中心);Dartmouth-Hitchcock Medical Center(达特茅斯-希区柯克医疗中心);Dartmouth College(达特茅斯学院)