Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into a fixed large number of visual tokens and then feed them into a Large Language Model (LLM). However, this design causes an excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency. While token pruning/merging methods do exist, they produce a single length output for each image and do not afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to-fine granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g. , adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2) M3 provides a framework for analyzing the granularity needed for existing datasets, where we find that COCO-style benchmarks only need around ~9 visual tokens to obtain accuracy similar to that of using all 576 tokens; (3) Our approach provides a foundation to explore the best trade-off between performance and visual token length at sample level, where our investigation reveals that a large gap exists between the oracle upper bound and current fixed-scale representations.
MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Alibaba Cloud Computing(阿里云计算);Hong Kong University of Science and Technology(香港科技大学)
Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models
The University of Tokyo(东京大学);S-Lab, Nanyang Technological University(南洋理工大学S实验室);Duke University(杜克大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);LY Corporation(LY公司);Tokyo University of Science(东京科学大学)
Promptception: How Sensitive Are Large Multimodal Models to Prompts?
Mohamed Bin Zayed University of Artificial Intelligence(莫罕默德·本·扎耶德人工智能大学);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院);Australian National University(澳大利亚国立大学)
SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation
National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);Zhejiang University(浙江大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community(多模态艺术投影研究共同体);Hong Kong University of Science and Technology(香港科技大学);University of Waterloo(滑铁卢大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peking University(北京大学);The Hong Kong Polytechnic University(香港理工大学);Waseda University(早稻田大学);University of Manchester(曼彻斯特大学)
SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);School of Computer Science(计算机科学学院);Baidu Inc.(百度公司);School of Computer Science, Wuhan University(武汉大学计算机学院)
Southeast University(东南大学);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, (Southeast University), Ministry of Education(新一代人工智能技术及其跨学科应用重点实验室);Nanyang Technological University(南洋理工大学)
LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Chinese Academy of Sciences(中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University(计算机学院、多媒体软件国家工程研究中心、人工智能研究院、多媒体与网络通信工程湖北省重点实验室、武汉大学)
MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
BIGAI(北京智源人工智能研究院);Shandong University(山东大学);University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Beijing Institute of Technology(北京理工大学)
MIBench: Evaluating LMMs on Multimodal Interaction
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China(中国人民大学人工智能学院,北京,中国);Beijing Key Laboratory of Research on Large Models(大型模型研究关键实验室);Beihang University, Beijing, China(北京航空航天大学,北京,中国);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室,上海,中国)
VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);School of Cyberspace Security, Sun Yat-sen University(中山大学网络空间安全学院)
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)