LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs
Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）
Shichu Sun, Yichen Zhang, Haolin Song, Zonghao Guo, Chi Chen, Yidan Zhang, Yuan Yao, Zhiyuan Liu, Maosong Sun
Visual encoding followed by token condensing has become the standard architectural paradigm in multi-modal large language models (MLLMs). Many recent MLLMs increasingly favor global native- resolution visual encoding over slice-based methods. To investigate this trend, we systematically compare their behavior on vision-language understanding and attention patterns, revealing that global encoding enhances overall capability but at the expense of greater computational overhead. To address this issue, we present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method, which can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding. The PVC approach consists of two key modules: (i) refined patch embedding, which supports flexible patch-size scaling for fine-grained visual model- ing, (ii) windowed token compression, hierarchically deployed across ViT layers to progressively aggregate local token representations. Jointly modulated by these two modules, a widely pretrained ViT can be reconfigured into an efficient architecture while largely preserving generality. Evaluated across extensive benchmarks, the transformed ViT, termed ViT-UHD, demonstrates competitive performance with MoonViT while reducing TTFT (time-to-first-token) by 2.4x, when developed within an identical MLLM architecture. Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x. We will release all code and checkpoints to support future research on efficient MLLMs.
LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer
Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；Alibaba Group（阿里巴巴集团）；University of Chinese Academy of Sciences（中国科学院大学）；The Chinese University of Hong Kong（香港中文大学）
Visual Position Prompt for MLLM based Visual Grounding
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent PCG（腾讯平台与内容事业群）；The Hong Kong University of Science and Technology（香港科技大学）；Dalian University of Technology（大连理工大学）
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Engineering, Westlake University（西湖大学工程学院）；University of Minnesota – Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Baidu Inc.（百度公司）；The University of Sydney（悉尼大学）；Amazon（亚马逊公司）；The Chinese University of Hong Kong（香港中文大学）
Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Images
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）；Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（pathology department, 国家癌症中心/国家癌症临床研究中心/癌症医院, 中国医学科学院和北京协和医学院）
ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（厦门大学多媒体可信感知与高效计算教育部重点实验室）；National University of Defense Technology（国防科技大学）
VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学）；Tencent Youtu Lab（腾讯优图实验室）
FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs
Shenzhen University（深圳大学）；Tsinghua University（清华大学）；The Hong Kong Polytechnic University（香港理工大学）；City University of Hong Kong（香港城市大学）；Sun Yat-sen University（中山大学）
LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education
Korea University（高丽大学）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）；Seoul Metropolitan Office of Education（首尔市教育厅）；Seoul National University（首尔国立大学）；Seoul National University of Education（首尔国立教育大学）
LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
The Chinese University of Hong Kong（香港中文大学）；City University of Hong Kong（香港城市大学）；The University of Hong Kong（香港大学）；Dongguan University of Technology（东莞理工学院）
Dynamic Pyramid Network for Efficient Multimodal Large Language Model
Beihang University（北航大学）；Shanghai AI Laboratory（上海人工智能实验室）；KAUST（卡塔尔大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Technical University Of Denmark（丹麦技术大学）；Nanjing University of Science and Technology（南京理工大学）；Peking University（北京大学）；Xiaohongshu Inc（小红书公司）
OrderChain: Towards General Instruct-Tuning for Stimulating the Ordinal Understanding Ability of MLLM
College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Transvascular Implantation Devices Research Institute and Liangzhu Laboratory（血管植入物研究机构和良渚实验室）；Ant Group（蚂蚁集团）；University of Notre Dame（圣母大学）；HKUST (Guangzhou)（香港科技大学（广州））
Speculative Decoding Reimagined for Multimodal Large Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室、中华人民共和国教育部、厦门大学）
Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
Mohamed bin Zayed University of AI（Mohamed bin Zayed人工智能大学）；Khalifa University（卡利法大学）；Michigan State University（密歇根州立大学）；Australian National University（澳大利亚国立大学）
LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
Ludwig Maximilian University of Munich（慕尼黑大学）；Munich Research Center, Huawei Technologies（华为技术有限公司慕尼黑研发中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）
WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image
Shenzhen University（深圳大学）；University of Nottingham Ningbo China（诺丁汉大学宁波分校）；City University of Hong Kong（香港城市大学）；Stanford University（斯坦福大学）；Hong Kong University of Science and Technology（香港科学与技术大学）
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；University of Washington（华盛顿大学）；Zhejiang University School of Medicine Sir Run Run Shaw Hospital（浙江大学医学院附属邵逸夫医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
