We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.
MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Tsinghua University(清华大学);Peking University(北京大学);Microsoft(微软公司)
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
Gaoling School of AI, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心,教育部);Ant Group(蚂蚁集团)
M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
Rochester Institute of Technology(罗切斯特理工学院);U.S. Naval Research Laboratory(美国海军研究实验室);Harvard Medical School(哈佛医学院);ByteDance(字节跳动);Meta AI;Meituan(美团);University of Science and Technology of China(中国科学技术大学);Shanghai Academy of AI for Science(上海人工智能科学研究院);Fudan University(复旦大学);University of Missouri - Kansas City(密苏里大学堪萨斯城分校);University of California - Davis(加利福尼亚大学戴维斯分校)
University of California, San Diego(加州大学圣地亚哥分校);Adobe Research(Adobe研究);The University of New South Wales(新南威尔士大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织的数据61)
HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
School of Advanced Interdisciplinary Sciences, UCAS(UCAS先进交叉科学学院);MAIS, CASIA(CASIA人工智能研究所);School of Artificial Intelligence, UCAS(UCAS人工智能学院);Centre for Artificial Intelligence and Robotics, HKISI-CAS(HKISI-CAS人工智能与机器人中心);FKLPRIU, School of Computer and Information Engineering, Xiamen University of Technology(厦门理工学院计算机与信息工程学院)
NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);Zhongguancun Academy(中关村学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tianjin University(天津大学);Nankai University(南开大学)
Discrete Prompt Tuning via Recursive Utilization of Black-box Multimodal Large Language Model for Personalized Visual Emotion Recognition
Graduate School of Information Science(信息科学研究生学校);Technology, Hokkaido University, Sapporo 060-0814, Japan(技术,北海道大学,札幌060-0814,日本);Office of Institutional Research, Hokkaido University, Sapporo 060-0808, Japan(机构研究办公室,北海道大学,札幌060-0808,日本);Faculty of Information Science(信息科学学院)
MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
Washington University in St. Louis(华盛顿大学圣路易斯分校);The University of British Columbia(不列颠哥伦比亚大学);Google Research(谷歌研究);Virginia Tech(弗吉尼亚理工大学);University of California, Davis(加州大学戴维斯分校);Sony AI(索尼人工智能);University of California, Berkeley(加州大学伯克利分校)
MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
The Ohio State University(俄亥俄州立大学);Imperial College London(伦敦帝国理工学院);The University of Hong Kong(香港大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Kyoto University(京都大学);Tencent AI Lab(腾讯AI实验室);Tomorrow Advancing Life(明天生命科技)
MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation
School of Computer Science and Technology, Tianjin University, Tianjin, China(天津大学计算机科学与技术学院);School of Software, Tsinghua University, Beijing, China(清华大学软件学院);School of Information Resource Management, Renmin University of China,Beijing, China(中国人民大学信息资源管理学院);School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China(上海交通大学人工智能学院);Baidu Inc., Beijing, China(百度公司)
Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);ShanghaiTech University(上海科技大学)
FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data
Nanjing University of Science and Technology(南京理工大学);National University of Singapore(新加坡国立大学);Institute of High-Performance Computing, A*STAR(新加坡高性能计算研究院(新加坡科研局下属机构))
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Insight-A: Attribution-aware for Multimodal Misinformation Detection
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Institute of Artificial Intelligence, Soochow University(苏州大学人工智能研究院);School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院)
CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
State Key Laboratory of Complex and Critical Software Environment, Beihang University(北京航空航天大学复杂关键软件环境国家重点实验室);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
Harbin Institute of Technology(哈尔滨工业大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济发展实验室);Chongqing Research Institute of HIT(重庆哈工大研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)