Evaluating instruction following capabilities for multimodal, multi-turn dialogue is challenging. With potentially multiple instructions in the input model context, the task is time-consuming for human raters and we show LLM based judges are biased towards answers from the same model. We propose MMMT-IF, an image based multi-turn Q$\&$A evaluation set with added global instructions between questions, constraining the answer format. This challenges models to retrieve instructions dispersed across long dialogues and reason under instruction constraints. All instructions are objectively verifiable through code execution. We introduce the Programmatic Instruction Following ($\operatorname{PIF}$) metric to measure the fraction of the instructions that are correctly followed while performing a reasoning task. The $\operatorname{PIF-N-K}$ set of metrics further evaluates robustness by measuring the fraction of samples in a corpus where, for each sample, at least K out of N generated model responses achieve a $\operatorname{PIF}$ score of one. The $\operatorname{PIF}$ metric aligns with human instruction following ratings, showing 60 percent correlation. Experiments show Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet, have a $\operatorname{PIF}$ metric that drops from 0.81 on average at turn 1 across the models, to 0.64 at turn 20. Across all turns, when each response is repeated 4 times ($\operatorname{PIF-4-4}$), GPT-4o and Gemini successfully follow all instructions only $11\%$ of the time. When all the instructions are also appended to the end of the model input context, the $\operatorname{PIF}$ metric improves by 22.3 points on average, showing that the challenge with the task lies not only in following the instructions, but also in retrieving the instructions spread out in the model context. We plan to open source the MMMT-IF dataset and metric computation code.
M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
Rochester Institute of Technology(罗切斯特理工学院);U.S. Naval Research Laboratory(美国海军研究实验室);Harvard Medical School(哈佛医学院);ByteDance(字节跳动);Meta AI;Meituan(美团);University of Science and Technology of China(中国科学技术大学);Shanghai Academy of AI for Science(上海人工智能科学研究院);Fudan University(复旦大学);University of Missouri - Kansas City(密苏里大学堪萨斯城分校);University of California - Davis(加利福尼亚大学戴维斯分校)
The Hong Kong University of Science and Technology(香港科技大学);ATH, Alibaba Group(阿里巴巴集团 ATH);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Northwestern Polytechnical University(西北工业大学);Zhejiang University(浙江大学)
MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping
University of California, San Diego(加州大学圣地亚哥分校);Johns Hopkins University(约翰霍普金斯大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Massachusetts Institute of Technology(麻省理工学院)
本文提出 MINT 任务分组策略,按多模态交互类型组织指令微调任务,以学习可迁移技能并减少任务干扰,在泛化与专业化间取得更优平衡。
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
CUHK MMLab(香港中文大学多媒体实验室);Waseda University(早稻田大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);SenseTime Research(商汤科技研究院);Shanghai AI Laboratory(上海人工智能实验室);CPII under InnoHK(创新香港旗下的CPII机构)
HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
School of Advanced Interdisciplinary Sciences, UCAS(UCAS先进交叉科学学院);MAIS, CASIA(CASIA人工智能研究所);School of Artificial Intelligence, UCAS(UCAS人工智能学院);Centre for Artificial Intelligence and Robotics, HKISI-CAS(HKISI-CAS人工智能与机器人中心);FKLPRIU, School of Computer and Information Engineering, Xiamen University of Technology(厦门理工学院计算机与信息工程学院)
University of California, San Diego(加州大学圣地亚哥分校);Adobe Research(Adobe研究);The University of New South Wales(新南威尔士大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织的数据61)
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Kyoto University(京都大学);Tencent AI Lab(腾讯AI实验室);Tomorrow Advancing Life(明天生命科技)
Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data
Department of Computer Science and Engineering, The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Translational Data Analytics Institute, The Ohio State University(转化数据分析研究所,俄亥俄州立大学);Department of Biomedical Informatics, The Ohio State University(生物医学信息学系,俄亥俄州立大学)
MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
The Ohio State University(俄亥俄州立大学);Imperial College London(伦敦帝国理工学院);The University of Hong Kong(香港大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
Washington University in St. Louis(华盛顿大学圣路易斯分校);The University of British Columbia(不列颠哥伦比亚大学);Google Research(谷歌研究);Virginia Tech(弗吉尼亚理工大学);University of California, Davis(加州大学戴维斯分校);Sony AI(索尼人工智能);University of California, Berkeley(加州大学伯克利分校)
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
Gaoling School of AI, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心,教育部);Ant Group(蚂蚁集团)
MLLMReID: Multimodal Large Language Model-based Person Re-identification
School of Computer Science and Engineering(计算机科学与工程学院);Beihang University(北京航空航天大学);State Key Laboratory of Virtual Reality Technology and Systems(虚拟现实技术与系统国家重点实验室)
MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
Northeastern University(东北大学);Nanjing University of Science and Technology(南京理工大学);University of Göttingen(哥廷根大学);Singapore University of Technology and Design(新加坡科技设计大学)
OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models
Arizona State University(亚利桑那州立大学);Clemson University(克莱姆森大学);Washington University in St. Louis(圣路易斯华盛顿大学);University of Notre Dame(诺特丹大学);Florida State University(佛罗里达州立大学);Rice University(里德大学);NVIDIA(英伟达);Mayo Clinic(梅奥诊所)
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo(滑铁卢大学);Georgia Institute of Technology(佐治亚理工学院);Hong Kong University of Science and Technology(香港科技大学);Google DeepMind(谷歌DeepMind)
MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Center for Brain-Inspired Computing Research (CBICR), Department of Precision Instrument, Tsinghua University(脑启发计算研究中心(CBICR)、精密仪器系,清华大学)